Re: email-based regression search server

[Phil Edwards <phil at jaj dot com>]

On Wed, Feb 05, 2003 at 01:57:23PM -0800, Janis Johnson wrote:
> There are several kinds of regressions requiring several kinds of tests.
> Some tests need only a compile, although perhaps with specific options,
> and some require running the resulting binary and perhaps comparing its
> output with expected output, or grepping the output of the compiler for
> specific messages.  An ICE for illegal code requires distinguishing
> between a compiler failure with a reasonable error message and one with
> an ICE message.

Oh, I understand all of that.  The search script has stubs in place for
equivalents of (most of?)  the dejagnu 'dg-do foo' keywords.  It's just
that doing anything besides 'assemble' requires a compiler driver, not
just a compiler, and I didn't preserve the drivers.  I can start doing
so, but I've got 6GB of cc1* already.  And the driver wouldn't be enough;
I'd also need to preserve libgcc to do many link tests, and maybe preserve
the specs file, and, and, and...


> Sometimes the test output is surprising, so it's
> important to capture that output so it can be examined.

As it stands, the output from the "breaking" compiler is one of the things
mailed back to the submitter.


As a prelude to your next paragraph, I should mention some background
information.  I have two scripts:  #1 does the actual searching.
The other, #2, handles the email interface to #1.  The "user interface"
(read, command-line options) of #1 is a superset of that offered by #2.

> Could your setup be modified to accept a test case plus a script to run
> it?  That script, like the one used by my regression search tools, could
> return 1 meaning to search later dates or 0 meaning to search earlier
> dates.

For the #1 script, certainly.  For the #2 script, no way in hell.  :-)

I realize it would be cool, but the thought of allowing anybody in the
world to email an arbitrary script to my machine and have it automatically
executed, just doesn't appeal to me.

Yes, I know I'm trying to keep the address nonpublished; someday it'll
leak out.  I know that I can make the procmail recipe check for a list
of valid users; email can be forged.  I can probably set up some kind of
chroot'd environment, but it would still bother me.  Doing this would,
eventually, not be different than simply giving out universal shell access.

Which leads me to this:  I /am/ willing to give shell access to the
well-known trusted GCC contributors who want to do more serious regression
hunting than is available through the email interface.


> It would also be useful to request running a test with compilers from a
> range of dates and getting back the output of all of the tests; this
> could be used to detect tests with inconsistent results (heisenbugs) if
> the results of a binary search don't seem reasonable.

Yep, see my reply to Joe.  This sounds more and more useful.


Phil

-- 
I would therefore like to posit that computing's central challenge, viz. "How
not to make a mess of it," has /not/ been met.
                                                 - Edsger Dijkstra, 1930-2002