This is the mail archive of the
gcc@gcc.gnu.org
mailing list for the GCC project.
Re: [BENCHMARK] Corrected gcc3.2 v gcc2.95.3 contest results
On Mon, Sep 23, 2002 at 08:30:21PM +1000, Con Kolivas wrote:
> Quoting Ingo Molnar <mingo@elte.hu>:
> > On Mon, 23 Sep 2002, Con Kolivas wrote:
> >
> > how many times are you running each test? You should run them at least
> > twice (ideally 3 times at least), to establish some sort of statistical
> > noise measure. Especially IO benchmarks tend to fluctuate very heavily
> > depending on various things - they are also very dependent on the initial
> > state - ie. how the pagecache happens to lay out, etc. Ie. a meaningful
> > measurement result would be something like:
>
> Yes you make a very valid point and something I've been stewing over privately
> for some time. contest runs benchmarks in a fixed order with a "priming" compile
> to try and get pagecaches etc back to some sort of baseline (I've been trying
> hard to make the results accurate and repeatable).
Well, run contest once, discard the results. Run it 3 more times, and
you should have started the second, third and fourth runs with similar initial conditions.
Or you could run the contest 3 times, rebooting between each run....
(automating that is a little harder, of course.)
IANAS, however.
--
Ryan Anderson
sometimes Pug Majere