SPEC2000 comparison of LLVM-3.2 and coming GCC4.8 on x86/x86-64

[Vladimir Makarov]

On 02/07/2013 11:09 AM, Richard Biener wrote:
> On Thu, Feb 7, 2013 at 4:26 PM, Vladimir Makarov <vmakarov@redhat.com> wrote:
>> I've add pages comparing LLVM-3.2 and coming GCC 4.8 on
>> http://vmakarov.fedorapeople.org/spec/.
>>
>> The pages are accessible by links named GCC-LLVM comparison, 2013, x86 and
>> x86-64 SPEC2000 under link named 2013. You can find these links at the
>> bottom of the left frame.
>>
>> If you prefer email for reading the comparison, here is the copy of page
>> accessible by link named 2013:
>>
>>
>> Comparison of GCC and LLVM in 2013.
>>
>> This year the comparison is done on coming *GCC 4.8* and *LLVM 3.2*
>> which was released at the very end of 2012.
>>
>> As usually I am focused mostly on the compiler comparison as
>> *optimizing* compilers on major platform x86/x86-64.  I don't consider
>> other aspects of the compilers as quality of debug information
>> (especially in optimizations modes), supported languages, standards
>> and extensions (e.g. OMP), supported targets and ABI, support of
>> just-in-time compilation etc.
>>
>> This year I did the comparison using following major options
>> equivalent with my point of view:
>>
>> o *-O0 -g, -Os, -O1, -O2, -O3, -O4* for LLVM3.2
>> o *-O0 -g, -Os, -O1, -O2, -O3, -Ofast -flto* for GCC4.8
> On the web-page you say that you use -Ofast -fno-fast-math (because
> that is what LLVM does with -O4).  For GCC that's equivalent to -O3
> (well, apart from that you enable -flto).  So you can as well say you
> tested -O3 -flto.
I guess -Ofast -fno-fast-math is not just -O3 but you are right it is 
pretty close.
> For 32bit you used -mtune=corei7 -march=i686 - did you disable
> CPU features like SSE on purpose?  Vectorization at -O3+ should
> have used those (though without -ffast-math FP vectorization is
> seriously restricted).
Yes, I did it on purpose.  Some 32-bit Linux distributions (e.g. Fedora) 
uses this architecture.  Another reason is that I'd like to see how good 
compilers work with fp stack (I got an impression that LLVM generates 
less fp stack regs shuffles, so I think we could improve regstack.c).  
Although it is a dying architecture and probably we should pay more 
attention to SSE architectures
> It would be nice to see -O3 -ffast-math vs. whatever LLVM equivalent
> is available.
That was my first intention.  Unfortunately, a few SPECFP tests do not 
generate expected results (and SPEC fails).

> Also note that for SPEC -funroll-loops helps GCC (yes ... we don't
> enable that by default at -O3, we probably should).
I should try this too.  My intention was to use most commonly known and 
used options with the point of view of average GCC user.  I believe that 
GCC has more potential to improve code for specific tests by advanced 
user as it has much more options and parameters to control the 
optimizations
> I don't know whether LLVM with -O4 creates fat objects as we do
> (you can link them without -flto).  If not, then for compile-time
> you should use -fno-fat-lto-objects.
Please, see comments above.
>    Does LLVM parallelize the
> LTO link stage?
No, I don't think so.
>    If so you should compare with -flto=jobserver
> or -flto=number-of-available-cores.  If not you should compare with
> -flto-partition=none (that will save some I/O and processing time).
I wanted to wrote about it.  LTO is well parallelized.  But I did not do 
this as my scripts uses CPU time.
> As a general note - we don't pay much attention to SPEC 2000
> performance these days but instead look at SPEC CPU 2006 ...
I'd like to but it still requires too much CPU time even on fastest 
machines.  Besides IMHO, some SPEC2006 tests are too memory bound and 
does not see many optimizations potential.  But may be next time.

>   
> Thanks for the comparison!
> Richard.
Richard, probably I am wrong, but I felt your frustration a bit. The 
bechmarking is evil.  Every time I post some benchmark results, I always 
see comments about benchmark credibility, wrong comparisons (used 
options) etc.  There is always some true in these comments. But I still 
believe they can help us too  see where we stand and where we are moving.

I am also a believer that GCC is much better and refined, and more 
reliable compiler working on evenly good on practically any platform and 
I prefer to work on GCC not LLVM (e.g. I thought about doing some RA 
work on LLVM long ago as their old RA was quite patetic but I preferred 
to work on GCC instead).

And thanks for the comments.