Re: Benchmarking (was Re: [RFC] Moving apart SGI extensions)

[Paolo Carlini <pcarlini at unitus dot it>]

Hi Phil,

sorry for the delay answering your message.

Phil Edwards wrote:

> On Thu, Dec 27, 2001 at 10:46:46AM +0100, Paolo Carlini wrote:
> > (I volunteer to work next on the benchmarking framework but I will need some
> > help for this... Phil?)
>
> The closest I got to doing any sort of benchmarking for libstdc++-v3 was
> hacking around with mkcheck.in, and the timing code wasn't even mine,
> I think.  :-)

This is a starting point! I could'nt even imagine that timing code was already present, ready to
use!

> My initial thoughts were to have a Stopwatch class in testsuite_hooks.h.
> Such a class (whatever its name) is a common extension in many commercial C++
> libraries, especially when platform-specific timing routines are possible.
>
>     Stopwatch   ticker;
>     ....
>     ticker.start();
>     a_string_instance.push_back (half a billion characters);
>     ticker.stop();
>     double clock_ticks = ticker.elapsed();
>
> etc, etc.
>
> The Stepanov tests also use some kind of timer.  I seem to have mislaid
> my copy of the source code, however.  There used to be something in the repo:
>
>     acorn 80% locate -i stepa
>     /home/pme/Repositories/GCC/gcc/gcc/testsuite/g++.old-deja/g++.other/Attic/stepanov_v1p2.C,v
>     acorn 81%

Yes. I remember Mark Mitchell put it in when he devised a cute patch which suddenly improved the
abstraction index noticeably on every platform... I wanted to track it, with the test failing as
soon as it is above 1.1 or something. Unfortunately, due to some problems on some
not-so-cleanly-structured ;-) backends (Mips probably) he add to revert the patch. Then also the
benchmark itself gone due to some copyright problems...
Anyway...
If you have managed to found the test and also the machinery used at the time (just a few days) to
actually time it this could be also an excellent starting point!

> But I'm sure he's constantly tweaking the tests, so that old copy probably
> wouldn't help us much anyhow.

Not really, to my best knowledge. I have here the very same version, that is the same also
available from a well know recently-buyed C++ compilers firm ;-)

> (I wonder if we can't use gprof to our advantage somehow.  We need something
> to analyze time spent in code blocks, gprof's raison d'etre is analyzing time
> spent in code blocks, hmmm.  I have no idea how to automate this, though.)

You know some of such things far better than me. My feeling is the approach used at the time for
Stepanov would be a very nice starting point. What do you think?

Cheers,
Paolo.