Re: [RFC] Our release cycles are getting longer

[David Carlton <david dot carlton at sun dot com>]

On Tue, 23 Jan 2007 17:54:10 -0500, Diego Novillo <dnovillo@redhat.com> said:

> So, I was doing some archeology on past releases and we seem to be
> getting into longer release cycles.

Interesting.

I'm a GCC observer, not a participant, but here are some thoughts:

As far as I can tell, it looks to me like there's a vicious cycle
going on.  Picking an arbitrary starting point:

1) Because lots of bugs are introduced during stage 1 (and stage 2),
   stage 3 takes a long time.

2) Because stage 3 takes a long time, development branches are
   long-lived.  (After all, development branches are the only way to
   do work during stage 3.)

3) Because development branches are long-lived, the stage 1 merges
   involve a lot of code.

4) Because the stage 1 merges involve a lot of code, lots of bugs are
   introduced during stage 1.  (After all, code changes come with
   bugs, and large code changes come with lots of bugs.)

1) Because lots of bugs are introduced during stage 1, stage 3 takes a
   long time.


Now, the good news is that this cycle can be a virtuous cycle rather
than a vicious cycle: if you can lower one of these measurements
(length of stage 3, size of branches, size of patches, number of
bugs), then the other measurements will start going down.  "All" you
have to do is find a way to mute one of the links somehow, focus on
the measurement at the end of that link, and then things will start
getting better.

It's not obvious to what the best way is to do that, but here are some
ideas.  Taking the links one by one:

1: Either fix bugs faster, or release with more bugs.

2: Artificially shorten the lifespan of development branches somehow,
so that big branches don't appear during stage 3.

3: Throttle the size of patches: don't let people do gigantic
merges, no matter the size of the branch.

4: Don't have buggy code in your branches: improve code quality of
development branches somehow.


I'm not optimistic about breaking either the link 1 or link 2.  The
first alternative in link 1 is hard (especially without a strong
social contract), and the second alternative in link 1 is, to say the
least, distasteful.  Link 2 is similarly hard to fix without a strong
social contract.  So I would focus on either link 3 or link 4.


For link 3, you'd change the rules to alternate between stage 1 and
stage 3 on a fast basis (no stage 2 would be necessary): do a small
merge (of a portion of a branch, if necessary), shake out bugs, and
repeat.  Concretely, you could have two rules in GCC's development
process:

* Patches more than a certain size aren't allowed.

* No patches are allowed if there are more than X release-blocking
  bugs outstanding.  (For some small value of X; 0 is one
  possibility.)

With this, the trunk is almost always in a releasable state; you can
release almost whenever you want to, since you'd basically be at the
end of stage 3 every week, or every day, or every hour.  Moving to
these rules would be painful, but once you start making progress, I
bet you'd find that, for example, the pressures leading to long-lived
branches will diminish.  (Not go away, but diminish.)


For 4, you should probably spend some time figuring out why bugs are
being introduced into the code in the first place.  Is test coverage
not good enough?  If so, why - do people not write enough tests, is it
hard to write good enough tests, something else?  Is the review
process inadequate?  If so, why: are rules insufficiently stringent,
are reviewers sloppy, are there not enough reviewers, are patches too
hard to review?

My guess is that most or all of those are factors, but some are more
important than others.  My favorite tactic to decrease the number of
bugs is to set up a unit test framework for your code base (so you can
test changes to individual functions without having to run the whole
compiler), and to strongly encourage patches to be accompanied by unit
tests.


And, of course, you could attack both links 3 and 4 at once.


David Carlton
david.carlton@sun.com