LTO: Speedup -- some preliminary SPEC2000 results

Wed Oct 7 14:39:00 GMT 2009

On Wed, Oct 7, 2009 at 4:25 PM, Vladimir Makarov <vmakarov@redhat.com> wrote:
> Jan Hubicka wrote:
>>
>> So things seems to work now plus minus as expected.  I.e. LTO builds
>> seems similar to combined builds and whole-programs improves code size
>> quite noticeably.
>> Runtime results for gzip are pretty much unchanged, but that is
>> expected.  I am quite curoius about full SPEC run.
>>
> Before the fix (Jan's two latest patches), the lto results were
> disappointed.  In brief the results I checked SPEC2000 a week ago on
> lto branch LTO on Core I7  (-O3 vs -O3 -flto with optional
> -fwhole-program) were
>  o Usage of LTO made compiler 1.9 time slower (in cpu time) for
>   SPECInt2000 and 2.2 time for SPECFP2000 on x86 and x86_64.
>  o LTO generated 16-17% bigger code for Int2000 and 13-14% bigger for
> FP2000.
>  o There is 0.6% improvement for SPECFP2000 on x86 and 1% for
>   SPECInt2000 on x86_64 (only because of 20% improvement on vortex,
>   all other tests were actually worse than without LTO).
>  o No improvement for Int2000 on x86 and FP2000 on x86_64.
>  o 252.eon and 176.gcc crash compiler when LTO were used.
>
> With latest Jan's fixes, The results (for -O3 vs -O3 -flto
> -fwhole-program) are
>
> x86:
>  o Int2000:
>   - LTO crashes the compiler on vortex.  LTO generates
>     wrong code for vpr, gcc, perlbmk, and gap.
>   - Compiler is 1.85 times slower with LTO
>   - Average code size is almost 6% smaller:
>
>        4.615%          44287          46331 164.gzip
>       -3.145%         144101         139569 175.vpr
>        0.261%        1566926        1571009 176.gcc
>      -12.118%          12279          10791 181.mcf
>       11.130%         209956         233324 186.crafty
>      -29.735%         155358         109162 197.parser
>      -23.075%         497347         382585 252.eon
>        8.904%         552163         601327 253.perlbmk
>        1.516%         503006         510630 254.gap
>      -20.891%          47465          37549 256.bzip2
>       -3.047%         198365         192321 300.twolf
>       Average = -5.96236%
>
>    - Performance is improved almost by 4%
>
>      164.gzip    1668   1629  -2.33813%
>      181.mcf     5011   5020   0.17960%
>      186.crafty  2268   2277   0.39682%
>      197.parser  1928   1925  -0.15560%
>      252.eon     2477   2950  19.0957%
>      256.bzip2   1894   1956   3.2735%
>      300.twolf   2806   3026   7.84034%
>      GeoMean     2416   2509   3.84934%
>
>  o FP2000
>   - LTO generates wrong code for mgrid, applu, galgel, facerec,
>     fm3d, sxitrack, and apsi.
>   - Compiler is 2.1 times slower with LTO
>   - Average code size is almost 1.7% smaller:
>
>      -8.771%          27544          25128 168.wupwise
>       2.328%           9108           9320 171.swim
>       2.127%          18193          18580 172.mgrid
>       0.004%          76584          76587 173.applu
>      -5.938%         576270         542049 177.mesa
>      -2.046%         183667         179910 178.galgel
>     -10.635%          15881          14192 179.art
>     -16.292%          28812          24118 183.equake
>      -3.177%          67239          65103 187.facerec
>      10.989%         125273         139039 188.ammp
>      -0.735%          49137          48776 189.lucas
>      -0.856%        1144550        1134756 191.fma3d
>      11.457%         935941        1043168 200.sixtrack
>      Average = -1.65735%
>
>    - Performance is improved almost by 6%
>
>      168.wupwise    2349    3266  39.0379%
>      171.swim       3511    3529   0.51267%
>      177.mesa       1970    2008   1.92893%
>      179.art        7097    7293   2.76173%
>      183.equake     3844    4138   7.64828%
>      188.ammp       2423    2401  -0.90796%
>      189.lucas      2825    2718  -3.78761%
>      GeoMean        3144    3332   5.97964%
>   x86_64:
>  o Int2000:
>   - LTO crashes the compiler on gcc.  LTO generates
>     wrong code for vpr, perlbmk, gap, and vortex
>   - Compiler is 1.8 times slower with LTO
>   - Average code size is more than 8% smaller:
>
>        1.376%          49119          49795 164.gzip
>       -4.348%         158389         151503 175.vpr
>      -16.964%          14949          12413 181.mcf
>       12.875%         195234         220370 186.crafty
>      -29.519%         180780         127416 197.parser
>      -22.894%         521614         402197 252.eon
>        9.507%         645749         707141 253.perlbmk
>        6.550%         585164         623492 254.gap
>      -22.493%         660414         511866 255.vortex
>      -18.343%          55825          45585 256.bzip2
>       -5.295%         212727         201463 300.twolf
>      Average = -8.14068%
>
>    - Performance is improved by 2.1%
>
>      164.gzip     1804    1773  -1.7184%
>      181.mcf      3480    3460  -0.5747%
>      186.crafty   3397    3406   0.2649%
>      197.parser   1847    1803  -2.3822%
>      252.eon      4071    4537  11.4468%
>      256.bzip2    2197    2249   2.3668%
>      300.twolf    2878    3048   5.9068%
>      GeoMean      2688    2744   2.0833%
>
>  o FP2000
>   - LTO crashes the compiler on apsi.  LTO generates wrong code for
>     mgrid, applu, galgel, facerec, fm3d, sixtrack.
>   - Compiler is 2.1 times slower with LTO
>   - Average code size is 2.7% smaller:
>
>      27.674%          33902          43284 168.wupwise
>      -3.107%          15704          15216 171.swim
>      -0.685%          22929          22772 172.mgrid
>      -1.167%         103280         102075 173.applu
>      -8.346%         678724         622079 177.mesa
>      -4.304%         249773         239024 178.galgel
>     -25.801%          20375          15118 179.art
>     -28.805%          37514          26708 183.equake
>      -1.577%          76837          75625 187.facerec
>       1.570%         168235         170877 188.ammp
>      -1.168%          57271          56602 189.lucas
>      -0.940%        1276316        1264314 191.fma3d
>      10.949%        1106507        1227658 200.sixtrack
>     Average = -2.74672%
>
>    - Performance is improved almost by 6%
>
>      168.wupwise     2532   3708  46.4455%
>      171.swim        3740   3729  -0.2941%
>      177.mesa        2969   2946  -0.7746%
>      179.art         7278   7092  -2.5556%
>      183.equake      3978   4227   6.2594%
>      188.ammp        2490   2515   1.0040%
>      189.lucas       3886   3806  -2.0586%
>      GeoMean         3603   3812   5.8007%
>
> LTO is quite promising.  Actually it is in line or even better with
> improvement got from other compilers (pathscale is the most convenient
> compiler to check lto separately: lto gave there upto 5% improvement
> on SPECFP2000 and 3.5% for SPECInt2000 making compiler about 50%
> slower and generated code size upto 30% bigger).  LTO in GCC actually
> results in significant code reduction which is quite different from
> pathscale.  That is one of rare cases on my mind when a specific
> optimization works actually better in gcc than in other optimizing
> compilers.  So congratulation to all people who worked on LTO!
>
> I think the biggest winner of LTO will be big C++ programs (eon shows
> that).  Additional optimizations (like devirtualization) could improve
> that results even more.  I think the next big thing would be using
> subtarget-specialized functions.

Note that there are daily runs for SPEC2000 and SPEC2006 on
x86_64 with -flto (and now -fwhopr) beyond gcc.opensuse.org.

SPEC2000 all compile and run successfully for me with -flto
with the exception of gcc which is non-conforming C code.

SPEC2006 is a different story, a bunch of tests do not have
enough memory to compile, another bunch miscompare or
crash.

Note that today we had additional breakage due to IPA-SRA,
after that is fixed results should look a lot better.

My performance obvservations before Honzas patch are
disappointing as well - just some minor speedups / slowdowns.

Richard.