povray: a benchmark with mainline, tree-ssa, and icc

[Scott Robert Ladd <coyote at coyotegulch dot com>]

Hello.

Given some questions posed in the "Contributing tree-ssa to mainline" 
thread, I decided to run a little benchmark.

The target of the benchmark is povray, a well-known ray-tracing program 
that is also getting some heavy use in my household at the moment. I 
pulled fresh source code for version 3.50c from their web site at 
http://www.povray.org. I used the official povray "benchmark.pov" with 
the settings in "benchmark.ini".

I ran these tests on a 2.8GHz Pentium 4 (Northwood) with 512MB RAM, 
under Linux 2.6.0, using mainline and tree-ssa code pulled form CVS mere 
hours ago. I also included results for Intel C++ 8.0, as a matter of 
comparison.

All times are in minutes:seconds, as reported by povray's internal 
timer. Switches used were:

gcc: -O3 -march=pentium4 --finline-functions
     -ffast-math -fomit-frame-pointer -funroll-loops
     -fexpensive-optimizations -malign-double
     -foptimize-sibling-calls -minline-all-stringops
     -Wno-multichar

icc: -O3 -xN -tpp7 -ipo

The GCC switches are the default defined by povray, with the exception 
of my changing the -march switch to specify the Pentium 4. I chose the 
ICC switches myself.

A drum-roll, please.

               compile  benchmark
                time      time
              --------  ---------
gcc mainline    1:43      7:59
gcc tree-ssa    2:40      7:24
icc 8.0         1:53      5:50


I don't find these numbers comforting.

Tree-ssa is substantially slower at compiling than mainline, although it 
does produce slightly faster code.

The ICC numbers speak for themselves. Yes, I verified the output as correct.

CAVEAT: This is only *one* platform, albeit a popular one, and only one 
benchmark. I'm going to run a full set of tests in the next few days, 
all on real world code.

Is povray a good benchmark? I think so, at least in terms of the kind of 
work I do, which is mostly number crunching.

Some anecdotal observations: On a DSP application for one of my clients, 
ICC is consistently 20% faster than GCC; tree-ssa offers a 5% 
improvement over mainline. Alas, the application is under NDA, so I 
can't provide more details. I do know that, with a very few exceptions, 
the povray numbers above reflect the relative performance of these 
compilers on my projects.

I realize some people do not see ICC as a valid competitor... however, I 
don't see why GCC can't at least approach ICC's level of performance.

At this point, tree-ssa is a marginal improvement over mainline in code 
quality, and a significant regression in terms of compile speed. Those 
who know tree-ssa may have some insight into whether or not more 
optimization can be squeezed out while also improving compile time.

..Scott

--
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing