This is the mail archive of the
gcc-patches@gcc.gnu.org
mailing list for the GCC project.
Re: fix kennerism
- From: "Joseph S. Myers" <jsm28 at cam dot ac dot uk>
- To: Richard Kenner <kenner at vlsi1 dot ultra dot nyu dot edu>
- Cc: <rth at redhat dot com>, <gcc-patches at gcc dot gnu dot org>
- Date: Sun, 30 Dec 2001 21:15:17 +0000 (GMT)
- Subject: Re: fix kennerism
On Sun, 30 Dec 2001, Richard Kenner wrote:
> I saw failures, but they looked pre-existing to me.
"looked pre-existing" is not a proper way of doing regression tests. You
should bootstrap and run tests on an unmodified FSF GCC tree, keep the
test output or the .sum files, apply a single patch (not multiple patches
at once) and again bootstrap and run tests (with exactly the same
configure options, on exactly the same host/build/target), and compare the
results of the two runs. You may sometimes find that there are unrelated
tests that fail or pass at random in your environment (based on observing
this in variation of results for unmodified GCC over time), and may then
list such tests that appeared to regress but claim that they are just
random failures, but should do a proper comparison of test results that
shows up any failures, random or otherwise.
The following are a bad idea:
* Comparing against a memory of test results rather than actual previous
results.
* Testing multiple patches together in the tree in a single test run.
* Testing in a non-FSF GCC tree and not testing in an FSF GCC tree.
* Comparing with results from any tree other than the exact state of the
tree before you applied your patch (for example, results from a CVS tree
from a few hours earlier, if you have done a cvs update since then).
--
Joseph S. Myers
jsm28@cam.ac.uk