Re: fix kennerism

["Joseph S. Myers" <jsm28 at cam dot ac dot uk>]

On Sun, 30 Dec 2001, Richard Kenner wrote:

>     You should bootstrap and run tests on an unmodified FSF GCC tree, 
> 
> What does "unmodified" mean here?  Neither patch development nor testing take
> zero time and the FSF GCC tree changes during that time.  There's no way to
> run a test on what the tree will be once the patch is installed because
> there's no way to predict what that will be.

Take a tree T that is the FSF GCC tree as of time t, with no patches 
applied.  Test T.  Test T+P where P is a single patch.

> If, when I wake up, the run is fine, what that proves is that things would
> have worked had I checked in the patches the night before.  But, of course, I
> could not have since I didn't have the results of the run.  There has been no
> testing with the state of the tree at the time the testing has completed.
> 
> We all understand this and accept it as a necessary risk of our development
> style since there's no alternative.  That wasn't what went wrong here.

The risk because the checkin is into tree T' of a later time t' is
unavoidable.  A test run, in the circumstances you describe, will have
tested T and T+P, so that P is known to work relative to T (but may, of
course, still have bad interactions with changes between t and t').

>     * Testing multiple patches together in the tree in a single test run.
> 
> As a practical matter, I don't have a choice because I get one testing run
> per day and would like to be able to fix more than one thing per day.  But
> what's the risk here?  If only some of the patches were doing to be
> installed, I'd agree with you, but since the intent is to install them *all*
> if the test passes, what is being shortcut here?

A test run, for multiple patches, should most safely test the trees (T,
T+P1, T+P2, ...) or (T, T+P1, T+P1+P2, T+P1+P2+P3, ...) if they are to be
installed together.  One patch might fix a problem, which another patch
then causes to appear again; or one might cause a problem, which another
covers up.  But separate testing is more important where patches are being
submitted for approval than when they will be installed together
immediately afterwards.

>     * Testing in a non-FSF GCC tree and not testing in an FSF GCC tree.
> 
> What do you mean by a "non-FSF GCC tree"?  Again, since development and
> testing don't take zero time, you have a moving target and I don't know what
> timepoints you mean here.

For example, testing of a patch in Red Hat's or Apple's internal tree is
not sufficient.

>     * Comparing with results from any tree other than the exact state of
>     the tree before you applied your patch (for example, results from a
>     CVS tree from a few hours earlier, if you have done a cvs update since
>     then).
> 
> If I understand what you are saying here correctly, this is what *has to
> happen* unless you can somehow prevent anybody from making other changes to
> the tree while the testing is running.  So perhaps I don't understand your
> point here.  Can you try again?

You test T and T+P, and compare, rather than comparing results from T and
T'+P.  The risk that checkins between t and t' cause problems when the
patch is checked into T' is still inevitable.

-- 
Joseph S. Myers
jsm28@cam.ac.uk