This is the mail archive of the
gcc@gcc.gnu.org
mailing list for the GCC project.
Re: [BENCHMARK] Corrected gcc3.2 v gcc2.95.3 contest results
- From: Ingo Molnar <mingo at elte dot hu>
- To: Con Kolivas <conman at kolivas dot net>
- Cc: linux-kernel at vger dot kernel dot org, <gcc at gcc dot gnu dot org>
- Date: Mon, 23 Sep 2002 16:26:34 +0200 (CEST)
- Subject: Re: [BENCHMARK] Corrected gcc3.2 v gcc2.95.3 contest results
- Reply-to: Ingo Molnar <mingo at elte dot hu>
On Mon, 23 Sep 2002, Con Kolivas wrote:
> Agreed. There probably is no statistically significant difference in the
> different gcc versions.
>
> Contest is very new and I appreciate any feedback I can get to make it
> as worthwhile a benchmark as possible to those who know.
your measurements are really useful i think, and people like Andrew
started to watch those numbers - this is why at this point a bit more
effort can/should be taken to filter out fluctuations better. Ie. a single
fluctuation could send Andrew out on a wild goose chase while perhaps in
reality his kernel was the fastest. Running every test twice should at
least give a ballpart figure wrt. fluctuations, without increasing the
runtime unrealistically.
i agree that only the IO benchmarks are problematic from this POV - things
like the process load and your other CPU-saturating numbers look perfectly
valid.
obviously another concern to to make testing not take days to accomplish.
This i think is one of the hardest things - making timely measurements
which are still meaningful and provide stable results.
Ingo