Re: "ways of accomplishing ... the basic goal"

[Tom Lord <lord at emf dot net>]

      Do you have any actual example of this, where a branch merge
      wasn't tested with no regressions in the testsuite on three
      platforms as required?

I nowhere claimed that such things have taken place (nor do I claim
they have not -- I simply don't know).   I do claim that the process
is a bit "gamed" -- and I hope that that's taken in the light it's
intended (as stating an obvious thing from an outsider perspective
that it is impolitic for the players to say themselves).

But the informative part of your reply is welcome:

    The problem after merges is mainly _unknown_ regressions -
    regressions not shown up in the testsuite.  Many of these could
    perhaps be found by bulk builds of real world software - such as
    BSD packages/ports collections and GNU/Linux distributions - but
    doing such builds requires vastly more resources than a
    bootstrap/test of GCC.
    (<http://gcc.gnu.org/ml/gcc/2002-10/msg00503.html> reports that a
    bulk build of FreeBSD packages takes 24 hours on a cluster of 8
    Pentium-III/800 machines.


That timescale is probably too large for general-purpose acceptance
tests on the trunk or for acceptance tests on sub-trunk branches for
"trivial" changes.

I would think it is _not_ too large for "promotion tests" for
non-trivial sub-trunk branches.  In other words, if you have a
sub-trunk branch that merges to the trunk every few days, at most,
while you wouldn't want to use a 24 hour test as an acceptance test
for that branch, you _could_ use it as a test that must pass before
the branch says "Hey, we're ready to merge with the trunk, now."


    For effective reduction of regressions in GCC this way you'd need
    at least such a cluster dedicated to testing GCC daily - 

Well, yes and no.

Yes: I do believe that substantial testing farms are a need of free
software projects generally, and GCC in particular.  I fully recognize
that there is a practical challenge of building such farms
economically.

No: you don't need to test the GCC trunk that way daily if changes to
the trunk are better controlled.  You can instead test a branch that
way, less frequently than daily.


    for mainline and each branch tested.  Preferably more often than
    daily, to identify individual patches responsible, though to some
    extent - if some machines are e.g. dedicated to providing hourly
    copies of GCC for binary search - binary search could be used for
    each broken package to find which patch broke it, without building
    everything every hour.  And then you want some human filtering for
    the cases where the problem was in the code that broke, not in
    GCC.)

You're heading towards something very fundamental.

You have, on the one hand, achievable rates of automated testing.

You have, on the other hand, achievable rates of commits.

In general, the achievable rates of commits greatly exceeds that of
testing.

The CRM/HSM proposal says: deal with this by imposing constraints.

Pick a level of testing for the trunk, then limit the commit rate to
that rate of testing.

Then make branches to take up the slack in commit rates -- recurse.

And you have to be smart about the branching because some kinds of
commits require different kinds of testing from others.  You want
fast-paths and slow-paths.


-t