[PATCH 0/6] Convert gimple to a C++ class hierarchy

Wed Sep 4 01:08:00 GMT 2013

On Mon, 2013-09-02 at 13:44 +0200, Michael Matz wrote:
> Hi,
> 
> On Fri, 30 Aug 2013, David Malcolm wrote:
> 
> > Here's the result of a pair of builds of r202029 without and with the
> > patches, configured with --enable-checking=release, running "make", then
> > stripping debuginfo [1]
> > 
> > So the overall sizes of such binaries are essentially unchanged.
> 
> Yep, cool.
> 
> > Any suggestions on what to compile to compare performance?  By 60 
> > seconds, do you mean 60s for one TU, or do you mean a large build e.g. 
> > the linux kernel?
> 
> For one TU, so as to not measure any significant execve, as or IO 
> overhead.  Some people compile e.g preprocessed variants of some GCC 
> source file, myself I'm measuring such speeds since some years with 
> unchanged versions of the attached (big-code.c, several large functions 
> with arithmetics and one-deep loops) which can be customized 
> by hand to produce different largeness and kdecore.cc [1] (real world C++ 
> library code from 2009).
> 
> You can customize big-code.c to match some compile time goal by 
> commenting out some FUNC invocation for some types, or by changing the 
> body size of the functions by fiddling in the FUNC macro itself to invoke 
> more of the L2's or even L3's, but that's really slow.

Thanks.  Everyone seems to have their own benchmarking test files - I'd
like to gather those that I can into a common repository.  What license
is big-code.c under?  Presumably kdecore.cc is from a compile of KDE and
thus covered by the license of that (plus of all the headers that were
included, I guess).

I scripted compilation of each of big-code.c and kdecore.cc at -O3,
using the cc1plus from each of the builds described above, 10 times
(i.e. the build of r202029 without (control) and with the patches
(experiment), both configured with --enable-checking=release, running
"make", then stripping debuginfo).

I wrote a script to parse the "TOTAL" timevar data from the logs,
extracting the total timings for user, sys, and wallclock, and the ggc
memory total. I used some code from Python's benchmarking suite to
determine whether the observed difference is "significant", and to draw
comparative graphs of the data (actually, to generate URLs to Google's
Chart API).

There were no significant differences for these data between the cc1plus
with the patch and the cc1plus without.  The script uses Student's
two-tailed T test on the benchmark results at the 95% confidence level
to determine significance, but it also has a cutoff in which when the
averages are within 1% of each other they are treated as
"insignificant".  If I remove the cutoff, then the kdecore.cc wallclock
time is reported as *faster* with the patch (t=2.41) - but by a tiny
amount.

FWIW this benchmarking code can be seen at:
https://github.com/davidmalcolm/gcc-build/commit/4581f080be2ed92179bfc1bc12e0ba9e923adaae

Data and URLs to graphs follow:

kdecore.cc -O3: usr
      control: [63.01, 63.11, 62.9, 63.14, 63.22, 63.0, 63.21, 63.03, 63.07, 63.09]
   experiment: [62.94, 63.05, 62.99, 63.03, 63.25, 62.91, 63.14, 62.96, 63.0, 63.04]
Min: 62.900000 -> 62.910000: 1.00x slower
Avg: 63.078000 -> 63.031000: 1.00x faster
Not significant
Stddev: 0.09852 -> 0.10049: 1.0200x larger
Timeline: http://goo.gl/8P9mFr

kdecore.cc -O3: sys
      control: [5.85, 5.74, 5.91, 5.79, 5.79, 5.85, 5.81, 5.9, 5.83, 5.8]
   experiment: [5.8, 5.81, 5.88, 5.88, 5.77, 5.87, 5.69, 5.8, 5.75, 5.84]
Min: 5.740000 -> 5.690000: 1.01x faster
Avg: 5.827000 -> 5.809000: 1.00x faster
Not significant
Stddev: 0.05229 -> 0.06154: 1.1769x larger
Timeline: http://goo.gl/xfWoUL

kdecore.cc -O3: wall
      control: [69.01, 69.01, 68.94, 69.03, 69.13, 68.99, 69.18, 69.07, 69.01, 69.01]
   experiment: [68.88, 68.97, 68.99, 69.03, 69.12, 68.88, 68.96, 68.86, 68.85, 68.99]
Min: 68.940000 -> 68.850000: 1.00x faster
Avg: 69.038000 -> 68.953000: 1.00x faster
Not significant
Stddev: 0.07052 -> 0.08616: 1.2217x larger
Timeline: http://goo.gl/Pf9BL8

kdecore.cc -O3: ggc
      control: [988589.0, 988591.0, 988585.0, 988582.0, 988584.0, 988589.0, 988585.0, 988582.0, 988584.0, 988586.0]
   experiment: [988593.0, 988585.0, 988589.0, 988585.0, 988591.0, 988588.0, 988591.0, 988585.0, 988589.0, 988590.0]
Mem max: 988591.000 -> 988593.000: 1.0000x larger
Usage over time: http://goo.gl/4pxB8Y

big-code.c -O3: usr
      control: [60.06, 60.31, 60.33, 60.29, 60.28, 60.26, 60.41, 60.28, 60.29, 60.29]
   experiment: [59.9, 60.14, 60.27, 60.33, 60.22, 60.33, 60.31, 60.33, 60.16, 60.26]
Min: 60.060000 -> 59.900000: 1.00x faster
Avg: 60.280000 -> 60.225000: 1.00x faster
Not significant
Stddev: 0.08781 -> 0.13360: 1.5215x larger
Timeline: http://goo.gl/gkFbgi

big-code.c -O3: sys
      control: [1.32, 1.37, 1.34, 1.41, 1.35, 1.34, 1.29, 1.36, 1.33, 1.35]
   experiment: [1.38, 1.35, 1.32, 1.32, 1.33, 1.33, 1.34, 1.32, 1.41, 1.35]
Min: 1.290000 -> 1.320000: 1.02x slower
Avg: 1.346000 -> 1.345000: 1.00x faster
Not significant
Stddev: 0.03169 -> 0.02953: 1.0731x smaller
Timeline: http://goo.gl/DPijei

big-code.c -O3: wall
      control: [61.48, 61.78, 61.78, 61.8, 61.74, 61.71, 61.81, 61.73, 61.73, 61.74]
   experiment: [61.39, 61.58, 61.7, 61.75, 61.65, 61.76, 61.76, 61.76, 61.67, 61.72]
Min: 61.480000 -> 61.390000: 1.00x faster
Avg: 61.730000 -> 61.674000: 1.00x faster
Not significant
Stddev: 0.09393 -> 0.11587: 1.2337x larger
Timeline: http://goo.gl/0WcMNl

big-code.c -O3: ggc
      control: [575294.0, 575294.0, 575294.0, 575294.0, 575294.0, 575294.0, 575294.0, 575294.0, 575294.0, 575294.0]
   experiment: [575294.0, 575294.0, 575294.0, 575294.0, 575294.0, 575294.0, 575294.0, 575294.0, 575294.0, 575294.0]
Mem max: 575294.000 -> 575294.000: no change
Usage over time: http://goo.gl/jO0Enh