Re: Performance regression testing?

[Mike Stump <mrs at apple dot com>]

On Nov 28, 2005, at 6:21 PM, Hans-Peter Nilsson wrote:
> I've attached the work-in-progress so I don't have to get into
> detail about what it does :-) except noting that you'll see in
> gcc.sum something like:
>
> PASS: csibe -O1 runtime zlib-1.1.4:minigzip not slower than best
> PASS: csibe -O1 runtime zlib-1.1.4:minigzip not more than .1%  
> slower than best
> PASS: csibe -O1 runtime zlib-1.1.4:minigzip not more than 1% slower  
> than best
> PASS: csibe -O1 runtime zlib-1.1.4:minigzip not more than 10%  
> slower than best

Hum, I'd prefer that the output format be:

PERF: %f name of test

then have an analysis package crunch that into the above format.   
This way, one can _compare_ two arbitrary runs, which is a useful  
property.  The number can be thought of as time, such as the number  
of clock cycles, but we only reeally care that it starts at zero, and  
gets bigger as things go bad.  An aside database would contain  
standard deviations, variances and so on, if one wanted to fold that  
into the comparisons.