Re: [BENCHMARK] Corrected gcc3.2 v gcc2.95.3 contest results

[Bill Davidsen <davidsen at tmr dot com>]

On Mon, 23 Sep 2002, Ingo Molnar wrote:

> 
> On Mon, 23 Sep 2002, Con Kolivas wrote:
> 
> > IO Full Load:
> > 2.5.38                  170.21          42%
> > 2.5.38-gcc32            230.77          30%
> 
> > This time only the IO loads showed a statistically significant
> > difference.
> 
> how many times are you running each test? You should run them at least
> twice (ideally 3 times at least), to establish some sort of statistical
> noise measure. Especially IO benchmarks tend to fluctuate very heavily
> depending on various things - they are also very dependent on the initial
> state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful
> measurement result would be something like:

Do note that the instructions for the benchmark suggest you boot single
user, which cuts down one problem, and since Con adopted my suggestion to
allow the user to set the location of the test file, I put the big file in
a filesystem which is formatted just before the test (I knew I'd find a
use for all that disk ;-) so that stays pretty constant.

The problem of memory size on the halfmem io is more serious, on a large
system the writes are all in memory, on a small system they cause
thrashing. I run in 256m for all tests just for this reason.

Not disagreeing with what you said, but the test is not inherently subject
to much jitter given care in running it.

-- 
bill davidsen <davidsen@tmr.com>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.