Re: Performance regression testing?

[Mike Stump <mrs at apple dot com>]

On Nov 28, 2005, at 4:38 PM, Mark Mitchell wrote:
> we require people to run regression tests for correctness, but that  
> we don't really have anything equivalent for performance.

My feeling is that we should have such a suite.  I'd favor a micro  
style, where we are measuring clock cycles (on machines that can  
expose them x86/v9), and then we re-run the suite (from the driver)  
enough to be sure we are near the minimum within some probability.    
The minimum then becomes the result.  The idea this would be a  
regression suite.

There is the other side of the coin, and that is macro testing, but  
generally speaking, we don't usually put macro tests in the testsuite  
(gcc/testsuite).  I think it would be good to collect such a suite  
and put it somewhere (ftp).  There are such suites available,  
BYTEmark (http://www.byte.com/bmark/bmark.htm) is just a random one  
that comes to mind.  Internally here, we have something called  
skidmarks that is a fast running, inner loop cut out of the hearth of  
the program type suite.  If people start taking up a collection for  
such a suite, we might be able to donate it (or parts of it).

I'd rather have someone running a performance regression tester often  
and not require people run the suite themselves (well, unless they  
are doing something like new ra, or tree-ssa).  A fast regression  
tester seems like it should be able to produce a new turn of the  
crank every 30 minutes (excluding ada/java/fortran), just doing an  
update, incremental make and install then run the suite.  This should  
be plenty enough resolution to spot the guilty parties.

The largest complication would seem to be a person to actually  
collect a suite that amuses them, and care for and feed a machine to  
do the running of the suite.