This is the mail archive of the
mailing list for the GCC project.
RE: SPEC95 performance comparison: 2.95.3 vs 3.0-20010525
- To: "Dan Nicolaescu" <dann at godzilla dot ICS dot UCI dot EDU>, <gcc at gcc dot gnu dot org>
- Subject: RE: SPEC95 performance comparison: 2.95.3 vs 3.0-20010525
- From: "M. Edward Borasky" <znmeb at aracnet dot com>
- Date: Sat, 26 May 2001 20:35:40 -0700
For all practical purposes, what these benchmarks show is that performance
is *independent* of compiler version -- there is no speed improvement or
penalty associated with "upgrading" from gcc 2.95.3 to 3.0-20010525 on these
How did I derive this nifty result, which may or may not be what the
developers who have worked long and hard on gcc 3.0 want to hear? Briefly, I
took each benchmark individually, computed the ratio of the gcc 2.95.3 time
to the 3.0 time and computed the geometric mean of the ratios. A ratio less
than 1 means that 2.95.3 is faster than 3.0 and a ratio greater than 1 means
that 3.0 is faster than 2.95.3. Here's the table:
Compiled with Compiled with
Benchmark gcc 2.95.3 gcc 3.0-20010525 Ratio
099.go-base 279 282 0.99
124.m88ksim-base 168 162 1.04
129.compress-base 181 176 1.03
130.li-base 165 168 0.98
132.ijpeg-base 205 215 0.95
134.perl-base 122 124 0.98
147.vortex-base 219 241 0.91
099.go-peak 283 276 1.03
124.m88ksim-peak 144 142 1.01
129.compress-peak 174 177 0.98
130.li-peak 171 171 1.00
132.ijpeg-peak 156 156 1.00
134.perl-peak 119 119 1.00
147.vortex-peak 218 239 0.91
Geometric Mean 0.99
You might have to play with tab settings to get these to line up in columns
correctly. I could go further, at the cost of some more sophisticated math,
to compute the 95% confidence limits for this mean, but it won't change the
fact that the average is very close to 1. Now, does anyone have any floating
point benchmarks they want to compare? Maybe even though these tests show no
improvement, for other types of workload there might be one ...
M. Edward (Ed) Borasky, Chief Scientist, Borasky Research
If there's nothing to astrology, how come so many famous men were born on
> -----Original Message-----
> From: Dan Nicolaescu [mailto:dann@godzilla.ICS.UCI.EDU]
> Sent: Saturday, May 26, 2001 3:25 PM
> To: email@example.com
> Subject: SPEC95 performance comparison: 2.95.3 vs 3.0-20010525
> I saw that there is some interest in comparing the performance of GCC
> 3.0 to previous versions, so I tried to see how it compares to
> GCC-2.95.3 on the SPECINT95 benchmarks.
> The results are here:
> 2.95.3 3.0-20010525
> Base Peak Base Peak
> Benchmarks Run Time Run Time Run Time Run Time
> ------------ -------- -------- -------- --------
> 099.go 279 283 282 276
> 124.m88ksim 168 144 162 142
> 129.compress 181 174 176 177
> 130.li 165 171 168 171
> 132.ijpeg 205 156 215 156
> 134.perl 122 119 124 119
> 147.vortex 219 218 241 239
> The numbers represent the running time in seconds.
> A short description of the benchmarks is at the end of this message.
> (note that 126.gcc is missing, the cc1 compiled with 3.0-20010525
> segfaults at runtime)
> The benchmarks are compiled with 2 sets of options:
> for base: -O3
> for peak: -O3 -mcpu=v8 -mtune=ultrasparc
> Suggestions for other options are welcome...
> (I should use -mcpu=v8 in the base configuration too.
> Why is the default instruction set v7? I doubt that there are many v7
> machines still around...)
> So for the base config li, ijpeg and vortex are worse for 3.0 in the
> base config, compress and vortex are worse for the peak config.
> No fancy options where used to when compiling the compilers: just
> The machine: SUN Blade 100
> Processor: 500MHz UltraSPARCIIe
> RAM: 1GB
> OS: Solaris 2.8
> The compilers and the benchmarks are on the local disk.
> The machine is in multi-user mode.
> I could also run the SPEC95 floating point benchmarks and report
> those results too.
> I have SPEC2000 too....
> Benchmark Descriptions:
> 099.go Game playing; artificial intelligence Plays the game Go
> against itself.
> 124.m88ksim Simulation Simulates the Motorola 88100 processor
> running Dhrystone and a memory test program.
> 126.gcc Programming & compilation Compiles pre-processed source
> into optimized SPARC assembly code.
> 129.compress Compression Compresses large text files (about 16MB)
> using adaptive Limpel-Ziv coding.
> 130.li Language interpreter Lisp interpreter.
> 132.ijpeg Imaging Performs jpeg image compression with various
> 134.perl Shell interpreter Performs text and numeric
> manipulations (anagrams/prime number factoring).
> 147.vortex Database Builds and manipulates three interrelated