Re: GCC trunk SPEC2000 performance

[Jan Hubicka <jh at suse dot cz>]

> You could always run SPEC after bootstrapping, but that is
> already being taken care of elsewhere.  Both SPEC95 and SPEC2000
> are run daily.  Besides, at 4 hours for SPEC95 and 10-12 hours
> for SPEC2000, that is an extremely expensive proposition for the
> automatic bootstrap tester.

I think we should shoot for the SPEC testers, not for the regression
tester to do the job. Regression testers are already bit overload
(SPEC testers as well)
> 
> > It may be interesting to use some statistics suite to postprocess the
> > results from Andreas/Diegos testers, find the gaps and put the links
> > in logs+notify the authors.  I tried to do this by hand for a while and
> >
> Agreed.  Furthermore, SPEC always exhibits a 1-5% variation from
> run to run.  To identify a true regression, you need to have a
> valley that lasts for at least 2-3 runs.  That's when you go to
> the first day of the valley and start looking at the diffs.

Does anyone have experinece with analyzing such records?  I didn't
visited my statistics classes very frequently, sorry :(
> 
> > We may think about using simplier testsuite targeted for various
> > optimizations.  Mark Lehnman and I attempted to put something together
> > at benchmarks subdirectory, but that is not maintained for a while.
> >
> There's always nullstone (http://www.nullstone.com), but I don't
> have a license to it.

Problem is that in case we say someone "nullstone test XYZ has regressed
20% after your patch", it almost no information as the sources are not
available.  Same applies to SPEC and it works in practice, so still
better than nothing, definitly.

Honza
> 
> Diego.