This is the mail archive of the gcc@gcc.gnu.org mailing list for the GCC project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]

Re: gzip performance test


[ Re: tests ]
> There are some remarks I would like to make:
> 
> 1) You should try tio eliminate "machine noise": benchmarks should be run in 
> single user mode (avoids daemon activity) and with network modules unloaded 
> to avoid interrupts generated by network activity.   And don't forget to do a 
> sync before each run

It suffices to run on a relatively inactive machine, run the test multiple
times, and discard outliers (that's why I reported the median and not the
average of five runs -- occasionally I got one test that was much slower
due to such activity, so I'd have one slow and four fast results).

> 2) Ideally each round orf benchmarks should use a glibc compiled by its own 
> compiler (and don't use the glibc shipped with your system since vendor could 
> be using more agressive compil parms than you).

I disagree (and of course this is only relevant for Gnu/Linux and maybe
Gnu/Hurd folks).  It's valid for all tests to use the same glibc, because we're
only looking for performance improvement or regression, we are not trying
to come up with numbers for formal publication.  And since the person who
installs gcc 3.0 in /usr/local/bin will probably continue to use the
vendor supplied glibc, what you advocate would give a distorted result:
end users will, at first, continue using their glibc and just update their
compiler.

> 3) You ran a gzip compied with gcc 2.95 and then a gzip compiled with gcc 3.0.
> Right?  Wrong.   Problem is processor temperature.

The effects you're seeing in this area probably get amplified because
you're doing 1), and thus starting with a relatively cold machine, plus
you're using typically crappy PC hardware.  Again, such effects are
handled by doing multiple runs, and the runs can be interleaved as well.




Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]