email-based regression search server

[Phil Edwards <phil at jaj dot com>]

On Fri, Jan 31, 2003 at 03:39:09PM -0800, Mike Stump wrote:
>On Friday, January 31, 2003, at 01:02 PM, Phil Edwards wrote:
>>I'm putting the finishing touches on an email-based binary search server.
>>One could send mail such as
>>
>> % mail -s 'regression hunt cc1plus 3.3 assemble -O2' foo@foo.foo < PRnnnn.cc
>>
>>and get back a message saying "It broke somewhere between the 12th and
>>the 13th."
> 
> Too cool for words!

The cool part I forgot to mention is that turnaround time is about 10
seconds, plus travel time for email.  Now that it's running, here are some
words anyhow.  Instructions, to be precise.

As a side effect of running the autocrasher, I have a couple month's worth
of cc1 and cc1plus, from daily builds of various branches.  (Right now they
number about 150 compilers.)  Good part is you get an answer quickly, bad
part is that it can't look further back than the start of December 2002.
Earlier ones got lost in a disk crash.

So this will start off as a "enh, may not be worth it" tool, and get more
useful as time goes on.

It's fairly trivial to do a binary search on those.  The resulting date
span should be taken with a metric buttload of salt, since (in the absence
of source) there's no telling when the sources were updated during the
course of a day.  Also we have fun with time zones.  The point is that the
results from this search server should be used as starting points for a
"real" binary search (using Janis' scripts), not as a replacement for one.
At best there's 24-hour granularity.

Instructions:

The body of the email is the source code.  No attachments, no .signatures,
just code.  If it uses any header files, you need to preprocess it first.

The assumption is that the code used to compile but now does not, and the
search will try to find when the compiler started exiting with a non-zero
value.  Not meeting that assumption will give amusingly useless results.

The subject line looks like this:

  Subject: regression hunt CC BRANCH DO-WHAT [OPTIONS] COMMENT

where

  CC is one of 'cc1' or 'cc1plus'
  BRANCH is one of '3.2', '3.3', or 'trunk'
  DO-WHAT is in theory one of a list of keywords, like the dejagnu
               hooks, but right now only 'assemble' makes any sense
  [OPTIONS] is a list of arbitrary compiler options, and yes, the
               square brackets are required even if OPTIONS is empty
  COMMENT is optional text, it just gets used in the reply Subject

Some examples of subject lines:

  regression hunt cc1plus trunk assemble []
  regression hunt cc1 trunk assemble [-pedantic] PR12345
  regression hunt cc1 3.3 assemble [-O3 -msoft-float] testcase from J.Random

Best to not use : or ! in the COMMENT text.

Do not anti-spam-munge your From: line or you won't get a response.
The hunter script is not that smart.  And I promise that the hunter won't
sell your email address to marketroids.

Finally, where to mail it to... ahem.  My procmail foo is quite good,
but I'd rather not leave the hunter address in the searchable archives.
Both spam and security worry me.  Anybody who wants to use it, just drop
me a line and I'll gladly give you the address.


Phil

-- 
I would therefore like to posit that computing's central challenge, viz. "How
not to make a mess of it," has /not/ been met.
                                                 - Edsger Dijkstra, 1930-2002