Re: compile time testsuite [Re: alias slowdown?]

["Richard Guenther" <richard dot guenther at gmail dot com>]

On 11/21/06, Benjamin Kosnik <bkoz@redhat.com> wrote:
> > Public monitoring would be more useful.  If you have working single-file
> > testcases that you want be monitored for compile-time and memory-usage
> > just contact me and I can add them to the daily tester
> > (http://www.suse.de/~gcctest/c++bench/).
>
> Hey Richard! Wow, this page is great: I'm amazed I didn't know about
> it. Thanks for the link. It's also nice to see a good database of
> libstdc++ performance testing results: this is really useful to me and
> I appreciate this resource being shared on the web.
>
> I do think that some kind of in-repository testsuite is necessary. I
> think this is important if people want to systematically tests
> compile-time performance *before* things are checked in, on their own
> machines.
>
> Over the long haul, I think that one of the safe generalizations about
> gcc infrastructure is that single points of failure should be avoided.
>
> So, I consider public monitoring complementary, but not something that
> obviates the real need for a consensus on what exactly are the things
> that are going to measured for compile-time regressions.

One problem with in-repository testing is that for compile-time performance
or memory usage you depend on a baseline or known good value.  At least
to automatically get a "FAIL" result here.  With recommended two runs,
one without, one with a patch we could do a post-processing script that
checks for regressions (hopefully not too often trapping on noise...).

(This is also one thing that makes the libstdc++ performance testsuite less
useful)

Richard.