Added LLVM 1.2 to nightly SPEC comparison runs

[Chris Lattner]

On Wed, 7 Apr 2004, Diego Novillo wrote:
> I have added LLVM v1.2 to the mix of baseline compilers we compare
> against on the nightly SPEC runs.

Cool!  :)

> Chris, I've used the flags '-O2 -Wl,-native-cbe' and '-O3
> -Wl,-native-cbe' for base and peak runs of LLVM.  I could use others,
> but the idea is to compare compilers with default optimizations.

LLVM ignores the -O* flags anyways, so the two runs should give identical
results. :)

> It's not easy to compare SPECfp because LLVM does not seem to have a
> fortran FE.

Quite true.  We haven't had anyone interested in doing the GCC work to
bring up the LLVM F77 front-end, so it hasn't been done yet. :(

> LLVM also seems much slower than tree-ssa.  Total build time for SPECint
> using tree-ssa is 362 seconds, compared to 797 seconds for LLVM.

How did you build LLVM?  By default it compiles in debug mode and
extensive assertion checking that are useful for development (think
"checking enabled"++).  If you build the tree with 'make
ENABLE_OPTIMIZED=1' you should get the binaries in the llvm/tools/Release
instead of the llvm/tools/Debug directory.  It is also worth checking to
make sure that the C front-end you are using is not compiled in debug
mode.

The other, perhaps more important, thing to remember is that you are
asking LLVM to compile the program *twice*: first with LLVM (optimizing it
and emitting a C file), then with GCC to compile the gigantic C file for
the whole program to native code.  These C files are often pretty big
(e.g., 966077 LOC and 42MB for 176.gcc), so that adds a substantial time
penalty to the compilation process.

Of course, there is a reason you're doing this: the native LLVM code
generators are not quite competitive yet (getting closer all of the time
though :), so using the C backend is the only real way to get interesting
performance numbers.  However, in a program development mode where compile
time matters, it would make a lot more sense to just use the LLVM JIT
compiler to get fast turn-around time to reduce the compile-build-test
cycle.

> What I found so far is that at -O2, tree-ssa tends to generate faster
> code except in eon (-8%) and, most notably, in art (-63%).

Not bad.  I'm suprised we actually do this well.  We haven't had anyone
that was seriously interested in performance, so the compiler is basically
completely untuned.  In particular, using GCC as a "backend" makes it
extremely difficult to characterize how to improve performance.  I suspect
that it would be relatively straight-forward to fix some of the most
glaring problems that come up.  I'll see if I can find some time before
the 1.3 release.  At the very least we should enable an interprocedural
pointer analysis or something.

One other thing:  If you're willing to update from LLVM CVS, I think
you'll see a nice performance improvement in the generated code. Due to a
restriction in the LLVM IR (which has now been fixed) LLVM 1.2 and before
were doing array address arithmetic with 64-bit integers, which was...
uhh... bad on X86.  :)  If you're rather wait until 1.3 is out, that's
also fine, it will probably be out in a couple of months or so.

> Detailed results for LLVM are available from the main SPEC2k test page
> at http://people.redhat.com/dnovillo/spec2000/tree-ssa-branch/

A couple of questions:

Do you have any idea why gcc, crafty, perlbmk and vortex are failing for
you?  They work fine for us, so I'd like to know if there is some sort of
bug that is triggering for you but not us or something.

Do you have any code size numbers?  Our inliner is currently tuned to be
pretty conservative, it would be interesting to see how size compares.

Is there any interest in compiling the output of LLVM with tree-ssa?  As
I mentioned before, LLVM is not exactly well served by using older GCC's
as it's backend: the emitted C code is very low-level.  It would be
interesting to see how LLVM+treessa stack up.

Also, can you change "gcc version 3.4-llvm 20030924 (experimental)" to
"LLVM 1.2"?  In terms of LOC, the non-c-front-end portion of LLVM amounts
to about 16x more code than we've added to the C front-end.  :)

Thanks!

-Chris

-- 
http://llvm.cs.uiuc.edu/
http://www.nondot.org/~sabre/Projects/