Re: GCC trunk SPEC2000 performance

[Jan Hubicka <jh at suse dot cz>]

> > Date: Thu, 20 Jun 2002 09:52:59 -0700
> > Cc: Dale Johannesen <dalej@apple.com>, Mark Mitchell <mark@codesourcery.com>,
> >    gcc@gcc.gnu.org, Geoff Keating <geoffk@geoffk.org>
> > From: Dale Johannesen <dalej@apple.com>
> 
> > On Wednesday, June 19, 2002, at 07:43 PM, David Edelsohn wrote:
> > 
> > > 	To avoid the argument that GCC 3.2 performance regressions were
> > > reported too late in the release cycle, does anyone know why the graphs on
> > > Andreas Jaeger's SPEC CPU2000 performance tracking page show:
> > 
> > Could the regression tester run some performance tests?  It would be
> > nice to catch these things early.
> 
> It could.  Would you like to suggest a patch (perhaps to
> contrib/regression/btest-gcc.sh), or a test procedure?  That would
> help the discussion, I think.
> 
> One of the most important design questions that need answering
> is "how much performance regression is too much?".

I am thinking about this idea for a while and for complex tests, like
spec, this is clearly dificult question, as majority of changes modify
the resuts by amount very close to statistic mistakes.

It may be interesting to use some statistics suite to postprocess the
results from Andreas/Diegos testers, find the gaps and put the links
in logs+notify the authors.  I tried to do this by hand for a while and
it did work.  Recently, unfortunately I don't have enought time and
energy to do the task.  I hope to find time to analyze this backward for
major perofmrance disasters during stage2 of development, as this has
shown to be effective, but tricky in last period.  I've done small study
of the patches that changes performance by more than one SPEC point in
the last period.  This is very rough image of what we may get if we do
it more seriously.

We may think about using simplier testsuite targeted for various
optimizations.  Mark Lehnman and I attempted to put something together
at benchmarks subdirectory, but that is not maintained for a while.
Similar attempt apepars to be bench++ suite that is more advanced,
perhaps we can use that one.

Honza
> 
> -- 
> - Geoffrey Keating <geoffk@geoffk.org> <geoffk@redhat.com>