Re: fix kennerism

[kenner at vlsi1 dot ultra dot nyu dot edu (Richard Kenner)]

    "looked pre-existing" is not a proper way of doing regression tests.

Clearly.  That shouldn't have had to have happened.

    You should bootstrap and run tests on an unmodified FSF GCC tree, 

What does "unmodified" mean here?  Neither patch development nor testing take
zero time and the FSF GCC tree changes during that time.  There's no way to
run a test on what the tree will be once the patch is installed because
there's no way to predict what that will be.

Consider the following: I develop and test a patch and update my tree to the
then-current FSF GCC tree, start off a test run, and go to sleep.

If, when I wake up, the run is fine, what that proves is that things would
have worked had I checked in the patches the night before.  But, of course, I
could not have since I didn't have the results of the run.  There has been no
testing with the state of the tree at the time the testing has completed.

We all understand this and accept it as a necessary risk of our development
style since there's no alternative.  That wasn't what went wrong here.

If I wake up and find the run has failed, there are two possible reasons for
the failure: either my change has a problem or some other change I imported
from the FSF GCC tree that day had a problem that showed up on my machine,
but not the one it was tested on.

If the failure is major, such as a bootstrap failure or a large number of
tests failing, I clearly have to get to the bottom of it before proceeding.
But if it's minor (a few tests failing), I have a judgement call to make as to
which of the two possibilities is the more likely.  I do this based on the
complexity of my change vs. the complexity of other changes made during that
day and using the traffic on the GCC lists as guidance.

In this case I jumped to the (incorrect) conclusion rather than spend more
time analyzing the results I got.  I'll try to be more careful in the future.

    The following are a bad idea:

    * Comparing against a memory of test results rather than actual previous 
    results.

Clearly.

    * Testing multiple patches together in the tree in a single test run.

As a practical matter, I don't have a choice because I get one testing run
per day and would like to be able to fix more than one thing per day.  But
what's the risk here?  If only some of the patches were doing to be
installed, I'd agree with you, but since the intent is to install them *all*
if the test passes, what is being shortcut here?

    * Testing in a non-FSF GCC tree and not testing in an FSF GCC tree.

What do you mean by a "non-FSF GCC tree"?  Again, since development and
testing don't take zero time, you have a moving target and I don't know what
timepoints you mean here.

    * Comparing with results from any tree other than the exact state of
    the tree before you applied your patch (for example, results from a
    CVS tree from a few hours earlier, if you have done a cvs update since
    then).

If I understand what you are saying here correctly, this is what *has to
happen* unless you can somehow prevent anybody from making other changes to
the tree while the testing is running.  So perhaps I don't understand your
point here.  Can you try again?