Performance regression testing?

[Mark Mitchell <mark at codesourcery dot com>]

We're collectively putting a lot of energy into performance improvements
in GCC.  Sometimes, a performance gain from one patch gets undone by
another patch -- which is itself often doing something else beneficial.
 People have mentioned to me that we require people to run regression
tests for correctness, but that we don't really have anything equivalent
for performance.

Clearly, performance testing is harder than correctness testing;
correctness is binary, while performance is a continuum.  Machine load
affects performance numbers.  It's reasonable to strive for no
correctness regressions, but introducing new optimizations is often
(always?) going to cause some code to perform worse.  If an optimization
was unsafe, then correctness concerns may require that we generate
inferior code.  So, it's hard problem.

The basic question I'm asking myself is: "Is there some pre check-in
testing we could do that would help make sure we're not backsliding?"
My goal is to make us aware of performance, without imposing anything
too burdensome.

As a strawman, perhaps we could add a small integer program (bzip?) and
a small floating-point program to the testsuite, and have DejaGNU print
out the number of iterations of each that run in 10 seconds.  The
results would show up on gcc-testresults automatically, and if we were
really eager, we could post the results along with our testing.  We
wouldn't have to treat inferior numbers as regressions in the same way
that we treat ordinary test failures, but maybe something like this
would help us to keep our eye on the ball.

Again, that's a strawman.  I'm just looking for suggestions about what
we might to do -- or even feedback that there's no need to do anything.

Thanks,

-- 
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com
(916) 791-8304