RE: SPEC95 performance comparison: 2.95.3 vs 3.0-20010525

["M. Edward Borasky" <znmeb at aracnet dot com>]

For all practical purposes, what these benchmarks show is that performance
is *independent* of compiler version -- there is no speed improvement or
penalty associated with "upgrading" from gcc 2.95.3 to 3.0-20010525 on these
benchmarks!

How did I derive this nifty result, which may or may not be what the
developers who have worked long and hard on gcc 3.0 want to hear? Briefly, I
took each benchmark individually, computed the ratio of the gcc 2.95.3 time
to the 3.0 time and computed the geometric mean of the ratios. A ratio less
than 1 means that 2.95.3 is faster than 3.0 and a ratio greater than 1 means
that 3.0 is faster than 2.95.3. Here's the table:

				Compiled with	Compiled with
Benchmark			gcc 2.95.3		gcc 3.0-20010525	Ratio
099.go-base			279			282			0.99
124.m88ksim-base		168			162			1.04
129.compress-base		181			176			1.03
130.li-base			165			168			0.98
132.ijpeg-base		205			215			0.95
134.perl-base		122			124			0.98
147.vortex-base		219			241			0.91
099.go-peak			283			276			1.03
124.m88ksim-peak		144			142			1.01
129.compress-peak		174			177			0.98
130.li-peak			171			171			1.00
132.ijpeg-peak		156			156			1.00
134.perl-peak		119			119			1.00
147.vortex-peak		218			239			0.91
Geometric Mean								0.99

You might have to play with tab settings to get these to line up in columns
correctly. I could go further, at the cost of some more sophisticated math,
to compute the 95% confidence limits for this mean, but it won't change the
fact that the average is very close to 1. Now, does anyone have any floating
point benchmarks they want to compare? Maybe even though these tests show no
improvement, for other types of workload there might be one ...
--
M. Edward (Ed) Borasky, Chief Scientist, Borasky Research
http://www.borasky-research.net  http://www.aracnet.com/~znmeb
mailto:znmeb@borasky-research.com  mailto:znmeb@aracnet.com

If there's nothing to astrology, how come so many famous men were born on
holidays?

> -----Original Message-----
> From: Dan Nicolaescu [mailto:dann@godzilla.ICS.UCI.EDU]
> Sent: Saturday, May 26, 2001 3:25 PM
> To: gcc@gcc.gnu.org
> Subject: SPEC95 performance comparison: 2.95.3 vs 3.0-20010525
>
>
> Hi!
>
> I saw that there is some interest in comparing the performance of GCC
> 3.0 to previous versions, so I tried to see how it compares to
> GCC-2.95.3 on the SPECINT95 benchmarks.
>
> The results are here:
>
> 			2.95.3             3.0-20010525
>                       Base       Peak       Base       Peak
>    Benchmarks     Run Time   Run Time   Run Time   Run Time
>    ------------   --------   --------   --------   --------
>    099.go              279        283        282        276
>    124.m88ksim         168        144        162        142
>    129.compress        181        174        176        177
>    130.li              165        171        168        171
>    132.ijpeg           205        156        215        156
>    134.perl            122        119        124        119
>    147.vortex          219        218        241        239
>
>
> The numbers represent the running time in seconds.
> A short description of the benchmarks is at the end of this message.
> (note that 126.gcc is missing, the cc1 compiled with 3.0-20010525
> segfaults at runtime)
>
> The benchmarks are compiled with 2 sets of options:
>
> for base: -O3
> for peak: -O3 -mcpu=v8 -mtune=ultrasparc
>
> Suggestions for other options are welcome...
> (I should use -mcpu=v8 in the base configuration too.
> Why is the default instruction set v7? I doubt that there are many v7
> machines still around...)
>
> So for the base config li, ijpeg and vortex are worse for 3.0 in the
> base config, compress and vortex are worse for the peak config.
>
> No fancy options where used to when compiling the compilers: just
> --prefix=BLAHBLAHBLAH
>
> The machine: SUN Blade 100
> Processor:   500MHz UltraSPARCIIe
> RAM:         1GB
> OS:          Solaris 2.8
> The compilers and the benchmarks are on the local disk.
> The machine is in multi-user mode.
>
> I could also  run the SPEC95 floating point benchmarks and report
> those results too.
> I have SPEC2000 too....
>
>
> Benchmark Descriptions:
>
> 099.go Game playing; artificial intelligence Plays the game Go
>         against itself.
>
> 124.m88ksim  Simulation Simulates the Motorola 88100 processor
>         running Dhrystone and a memory test program.
> 126.gcc  Programming & compilation Compiles pre-processed source
>         into optimized SPARC assembly code.
> 129.compress  Compression Compresses large text files (about 16MB)
>         using adaptive Limpel-Ziv coding.
> 130.li Language interpreter Lisp interpreter.
> 132.ijpeg  Imaging Performs jpeg image compression with various
>         parameters.
>
> 134.perl  Shell interpreter Performs text and numeric
>         manipulations (anagrams/prime number factoring).
> 147.vortex Database Builds and manipulates three interrelated
>         databases.
>
>
>
>