3.3-branch QA assessment

[Joe Buck <jbuck at synopsys dot com>]

I've been looking at our high-priority regressions and have come to a
somewhat disturbing conclusion.  Now, regressions aren't the whole story,
but based only on the regression data, it seems that 3.3 is in
substantially worse shape than either 3.2.x or 3.4, begging the question
of what the point of the 3.3 branch is.  (Please read to the end before
you flame).

There are 118 high-priority PRs.  One, about Java race conditions, doesn't
look like it is really a regression, leaving 117 that can be classified.
They break down as follows:

      1	    3.2.1/3.3/3.4 regression
      1	    3.2.1 regression
     25	    3.2/3.3/3.4 regression
     21	    3.2/3.3 regression
     20	    3.2 regression
      9	    3.3/3.4 regression
     20	    3.3 regression
     19	    3.4 regression

Let's say we ignore the existence of 3.3 (pretend that 3.4 is the next
major release).  Then any references to 3.3 above go away; we have

-: bugs that go away
+: new bugs

      1	    3.2.1/3.4 regression
      1	    3.2.1 regression -
     25	    3.2/3.4 regression
     41	    3.2 regression -
     28	    3.4 regression +

Interpretation: with the next major release, we close 1+41 regressions and add
28 new regressions, for a net improvement of 14 regressions.  Now, of
course the trunk hasn't had a huge amount of testing yet, but this
indicates that as of today, 3.4 looks less buggy than 3.2.2 will be,
even before any effort is made to stabilize it and clean it up.

Let's say we ignore the existence of 3.4.  Then we get.

      1	    3.2.1/3.3 regression
      1	    3.2.1 regression -
     46	    3.2/3.3 regression
     20	    3.2 regression -
     29	    3.3 regression +

Interpretation: with the next major release, we close 1+20 regressions and
add 29 new regressions, giving us 9 more regressions than we have now!
We would have to do substantial work on 3.3 to get the quality up.
Meanwhile, we have Matt Austern's numbers showing that it is substantially
slower.  Are we just going to release the thing because we are on track
to do so, come what may, no matter how bad it is?

It begs the question of who would want the 3.3 release.

I'm afraid folks are going to hate me, but I'm seriously suggesting just
dumping the 3.3 branch, unless someone can explain to me why I'm
completely wrong.  I've been in companies who have gone through the
experience of dumping a release when things like this happen; it is
painful and expensive, but sometimes it can be better than the
alternative.