Re: gzip performance test

[Jean Francois Martinez <jfm2 at club-internet dot fr>]

On Friday 25 May 2001 16:58, David Edelsohn wrote:
> I do not think that gzip is a very useful performance test.  It
> mainly is a question of how well GCC optimizes one loop.  The gzip part of
> the GCC 3.0 release criteria was more about code quality and not
> encoutering a regression, not about showing a large improvement.
>
> 	Using nbench / BYTEmark benchmark for powerpc-ibm-aix4.3.3.0
>
> 	GCC 2.95.3 gets:
>
> MEMORY INDEX        : 1.699
> INTEGER INDEX       : 2.205
> FLOATING-POINT INDEX: 3.583
>
> 	GCC 3.0 prerelease gets:
>
> MEMORY INDEX        : 1.748
> INTEGER INDEX       : 2.683
> FLOATING-POINT INDEX: 3.710
>
> 	IBM's XLC 3.6.6 gets:
>
> MEMORY INDEX        : 1.830
> INTEGER INDEX       : 2.294
> FLOATING-POINT INDEX: 3.960
>
>


There are some remarks I would like to make:

1) You should try tio eliminate "machine noise": benchmarks should be run in 
single user mode (avoids daemon activity) and with network modules unloaded 
to avoid interrupts generated by network activity.   And don't forget to do a 
sync before each run

2) Ideally each round orf benchmarks should use a glibc compiled by its own 
compiler (and don't use the glibc shipped with your system since vendor could 
be using more agressive compil parms than you).   Using same glibc flattens 
results and there is also the question of glibc compiled with old compiler 
could have a negative effect with code genearated by new compiler (for 
instance due tio register assignment).

3) You ran a gzip compied with gcc 2.95 and then a gzip compiled with gcc 3.0.
Right?  Wrong.   Problem is processor temperature.  Processors slow down when 
they heat  On a loop of seven Byte benchmarks in a row I have noticed some 
tests having a standard deviation of nearly 5% (thus about ten percent 
between slowest and fastest)  and some tests failed the statistical test 
performed by the Byte bechmark due to too strong variations.  The tests with 
strong variations were those who came immedately after  a floating point (ie 
heat intensive) test.  Box was a Dell desktop with a PIII/667, room was cool, 
nearly cold and despite this computer was slowing due to heat produced by 1 
minute of floating point activity.   So much for big names producing 
expensive but high quality computers.


				JFM