This is the mail archive of the
gcc@gcc.gnu.org
mailing list for the GCC project.
Re: email-based regression search server
On Wed, Feb 05, 2003 at 02:43:49PM -0500, Phil Edwards wrote:
> Two minor updates:
> On Mon, Feb 03, 2003 at 06:21:18PM -0500, Phil Edwards wrote:
>
> > The assumption is that the code used to compile but now does not, and the
> > search will try to find when the compiler started exiting with a non-zero
> > value. Not meeting that assumption will give amusingly useless results.
>
> It can now also look for the opposite situation: the input code was broken,
> but now compiles. (I don't know of an example.)
>
> Input test code that worked, then broke, then worked, then broke, etc, will
> naturally drive the hunter nuts. Like you'd expect with a binary search.
Phil,
There are several kinds of regressions requiring several kinds of tests.
Some tests need only a compile, although perhaps with specific options,
and some require running the resulting binary and perhaps comparing its
output with expected output, or grepping the output of the compiler for
specific messages. An ICE for illegal code requires distinguishing
between a compiler failure with a reasonable error message and one with
an ICE message. Sometimes the test output is surprising, so it's
important to capture that output so it can be examined.
Could your setup be modified to accept a test case plus a script to run
it? That script, like the one used by my regression search tools, could
return 1 meaning to search later dates or 0 meaning to search earlier
dates.
It would also be useful to request running a test with compilers from a
range of dates and getting back the output of all of the tests; this
could be used to detect tests with inconsistent results (heisenbugs) if
the results of a binary search don't seem reasonable.
I recently tried to automate making a script to run a regression test.
It was tied to my environment for binary searches, but if you're
interested, let me know and perhaps we can come up with something that
will work with your system.
Janis