Re: Benchmark suite...

[joel at OARcorp dot com]

On Wed, 21 Oct 1998, Michael Meissner wrote:

> On Wed, Oct 21, 1998 at 01:28:22PM +0200, Jan Hubicka wrote:
> > > 
> > > What do you think about the idea of adding gzip/bzip2 to the test suite?
> > It is one thinks I want to do today. Problem is that gzip itself
> > is long. I will have to dig out deflate code and make suitable test.
> > (using whole gzip or whole crafty is IMO bad, because compilation time
> > would increase significantly)
> > > It might also be useful to add code that currently isn't optimized as wel
> > > as it could be by the compiler: e.g. tail-recursive functions.  That could
> > > provide some incentive to implement better mechanisms in the compiler.
> > I've done naive implementations of recursive hanoi solver and quicksort,
> > wich is first of these. Especially quicksort shows VERY interesting behaviour.
> > Of course I will welcome other code too.
> > I also want to use dryrstoe benchmark and dig out suitable tests from byte
> > benchmarks.
> 
> This will break embedded environments which don't have filesystems or I/O in
> the general sense.  If this goes in, please put it under something like make
> benchmark, and NOT under make check.

Please don't forget that embedded systems generally have a small stack in
comparison to UNIX systems.  On UNIX systems, the stack can grow in
virtual memory.  This doesn't happen on embedded systems. 

I mention this because a heavily recursive benchmark is almost certainly
going to use a lot of stack space and not run on most embedded targets
even if it avoids file IO.

> As a side note, hanoi, etc. are part of the Stanford benchmark suite.  It is
> generally felt these are useless for today's apps.  In particular, they all fit
> in the L1 cache of today's modern chips (hell, spec 92 nearly fits in the L2
> cache nowadays).  Also when I was using the benchmark suite 9+ years ago to
> tune the 88k compiler, I found that the system clock (which for the 88k was 100
> times a second) was too coarse grained to be of any use -- one clock tick
> more/less would seriously upset the timing.

Another technique is to measure how many iterations of something you can
do in a specific period of time (usually at least a second).  This does
not solve the cache issue but does help on the timing issue.

--jel