Re: Automated regression testing

[Geoff Keating <geoffk at geoffk dot org>]

> Date: Mon, 19 Mar 2001 00:57:27 -0500 (EST)
> From: John Wehle <john@feith.com>
> Cc: gcc@gcc.gnu.org

> > Finally, note that the whole point of the regression tester is that it
> > does its very best to only send mail to the person who actually caused
> > the regression.
> 
> I agree that it's best if the nag message is limited to the person who
> actually caused the problem and if it's limited to only one message.
> However, it'd be nice if more platforms were covered and that might
> involve some tradeoffs.

I agree that more platforms should be covered, but I don't like the
idea of "tradeoffs".  Especially since it's really not that costly to
add enough hardware to the existing tester to allow all those cross
targets to be tested without affecting the tester's performance.  If
it wasn't for the fact that the software isn't written yet, I would be
trying to arrange for hardware now.

(The important bit of software that's now being written is the part
where multiple sets of test results are combined into a single mail
message.  I think this is essential.)

> I actually started to play with this in the interest of being able to
> do precommit testing of my own patches on more platforms in a less error
> prone way.  It'd be nice if the regression tester would tag the most
> recent code which is known to build and pass make check on X
> platforms

I did consider this, several times, but rejected it because of the
load it'd impose on the CVS server; tagging is a very expensive
operation because it changes every RCS file.  Instead, you can
grab the status file
<http://www.cygnus.com/~geoffk/gcc-regression/status.txt>, and do

cvs co -D `awk '/^Last-Pass:/ { print $2; } < status.txt` gcc-full

A lot of the regression tester's limitations are really caused by not
getting the support it needs from CVS.

> in addition to automatically checking in the expected results.

The current expected results, but not past ones, are available
from a URL near the one above.  I'd make historical data available if
there was demand.

>  I'd
> then like to run a script which checks out a local copy of the code,
> applies my patch, runs a local copy of the tester, and reports to me
> the results.
> 
> I think that developers are happy to do more precommit testing if it
> doesn't involve too much overhead on their part.

One of the projects I'm currently working on is a script which will
take a patch and validate it against the CVS tree and regression
tester results---although, in truth, it's likely to be less than 50
lines of shell script, so it's not like it's so much work that no-one
else could do it.

Just the machine time required to test against lots of targets is
significant overhead.

-- 
- Geoffrey Keating <geoffk@geoffk.org>