Re: Benchmarks of v2 (was Re: [PATCH 0/5] RFC: Overhaul of diagnostics (v2))

[Richard Biener <richard dot guenther at gmail dot com>]

On Tue, Oct 13, 2015 at 5:32 PM, David Malcolm <dmalcolm@redhat.com> wrote:
> On Thu, 2015-09-24 at 10:15 +0200, Richard Biener wrote:
>> On Thu, Sep 24, 2015 at 2:25 AM, David Malcolm <dmalcolm@redhat.com> wrote:
>> > On Wed, 2015-09-23 at 15:36 +0200, Richard Biener wrote:
>> >> On Wed, Sep 23, 2015 at 3:19 PM, Michael Matz <matz@suse.de> wrote:
>> >> > Hi,
>> >> >
>> >> > On Tue, 22 Sep 2015, David Malcolm wrote:
>> >> >
>> >> >> The drawback is that it could bloat the ad-hoc table.  Can the ad-hoc
>> >> >> table ever get smaller, or does it only ever get inserted into?
>> >> >
>> >> > It only ever grows.
>> >> >
>> >> >> An idea I had is that we could stash short ranges directly into the 32
>> >> >> bits of location_t, by offsetting the per-column-bits somewhat.
>> >> >
>> >> > It's certainly worth an experiment: let's say you restrict yourself to
>> >> > tokens less than 8 characters, you need an additional 3 bits (using one
>> >> > value, e.g. zero, as the escape value).  That leaves 20 bits for the line
>> >> > numbers (for the normal 8 bit columns), which might be enough for most
>> >> > single-file compilations.  For LTO compilation this often won't be enough.
>> >> >
>> >> >> My plan is to investigate the impact these patches have on the time and
>> >> >> memory consumption of the compiler,
>> >> >
>> >> > When you do so, make sure you're also measuring an LTO compilation with
>> >> > debug info of something big (firefox).  I know that we already had issues
>> >> > with the size of the linemap data in the past for these cases (probably
>> >> > when we added columns).
>> >>
>> >> The issue we have with LTO is that the linemap gets populated in quite
>> >> random order and thus we repeatedly switch files (we've mitigated this
>> >> somewhat for GCC 5).  We also considered dropping column info
>> >> (and would drop range info) as diagnostics are from optimizers only
>> >> with LTO and we keep locations merely for debug info.
>> >
>> > Thanks.  Presumably the mitigation you're referring to is the
>> > lto_location_cache class in lto-streamer-in.c?
>> >
>> > Am I right in thinking that, right now, the LTO code doesn't support
>> > ad-hoc locations? (presumably the block pointers only need to exist
>> > during optimization, which happens after the serialization)
>>
>> LTO code does support ad-hoc locations but they are "restored" only
>> when reading function bodies and stmts (by means of COMBINE_LOCATION_DATA).
>>
>> > The obvious simplification would be, as you suggest, to not bother
>> > storing range information with LTO, falling back to just the existing
>> > representation.  Then there's no need to extend LTO to serialize ad-hoc
>> > data; simply store the underlying locus into the bit stream.  I think
>> > that this happens already: lto-streamer-out.c calls expand_location and
>> > stores the result, so presumably any ad-hoc location_t values made by
>> > the v2 patches would have dropped their range data there when I ran the
>> > test suite.
>>
>> Yep.  We only preserve BLOCKs, so if you don't add extra code to
>> preserve ranges they'll be "dropped".
>>
>> > If it's acceptable to not bother with ranges for LTO, one way to do the
>> > "stashing short ranges into the location_t" idea might be for the
>> > bits-per-range of location_t values to be a property of the line_table
>> > (or possibly the line map), set up when the struct line_maps is created.
>> > For non-LTO it could be some tuned value (maybe from a param?); for LTO
>> > it could be zero, so that we have as many bits as before for line/column
>> > data.
>>
>> That could be a possibility (likewise for column info?)
>>
>> Richard.
>>
>> > Hope this sounds sane
>> > Dave
>
> I did some crude benchmarking of the patchkit, using these scripts:
>   https://github.com/davidmalcolm/gcc-benchmarking
> (specifically, bb0222b455df8cefb53bfc1246eb0a8038256f30),
> using the "big-code.c" and "kdecore.cc" files Michael posted as:
>   https://gcc.gnu.org/ml/gcc-patches/2013-09/msg00062.html
> and "influence.i", a preprocessed version of SPEC2006's 445.gobmk
> engine/influence.c (as an example of a moderate-sized pure C source
> file).
>
> This doesn't yet cover very large autogenerated C files, and the .cc
> file is only being measured to see the effect on the ad-hoc table (and
> tokenization).
>
> "control" was r227977.
> "experiment" was the same revision with the v2 patchkit applied.
>
> Recall that this patchkit captures ranges for tokens as an extra field
> within tokens within libcpp and the C FE, and adds ranges to the ad-hoc
> location lookaside, storing them for all tree nodes within the C FE that
> have a location_t, and passing them around within c_expr for all C
> expressions (including those that don't have a location_t).
>
> Both control and experiment were built with
>   --enable-checking=release \
>   --disable-bootstrap \
>   --disable-multilib \
>   --enable-languages=c,ada,c++,fortran,go,java,lto,objc,obj-c++
>
> The script measures:
>
> (a) wallclock time for "xgcc -S" so it's measuring the driver, parsing,
> optimimation, etc, rather than attempting to directly measure parsing.
> This is without -ftime-report, since Mikhail indicated it's sufficiently
> expensive to skew timings in this post:
>   https://gcc.gnu.org/ml/gcc/2015-07/msg00165.html
>
> (b) memory usage: by performing a separate build with -ftime-report,
> extracting the "TOTAL" ggc value (actually 3 builds, but it's the same
> each time).
>
> Is this a fair way to measure things?  It could be argued that by
> measuring totals I'm hiding the extra parsing cost in the overall cost.

Overall cost is what matters.   Time to build the libstdc++ PCHs
would be interesting as well ;)  (and their size)

One could have argued you should have used -fsyntax-only.

> Full logs can be seen at:
>   https://dmalcolm.fedorapeople.org/gcc/2015-09-25/bmark-v2.txt
> (v2 of the patchkit)
>
> I also investigated a version of the patchkit with the token tracking
> rewritten to build ad-hoc ranges for *every token*, without attempting
> any kind of optimization (e.g. for short ranges).
> A log of this can be seen at:
> https://dmalcolm.fedorapeople.org/gcc/2015-09-25/bmark-v2-plus-adhoc-ranges-for-tokens.txt
> (v2 of the patchkit, with token tracking rewritten to build ad-hoc
> ranges for *every token*).
> The nice thing about this approach is that lots of token-related
> diagnostics gain underlining of the relevant token "for free" simply
> from the location_t, without having to individually patch them.  Without
> any optimization, the memory consumed by this approach is clearly
> larger.
>
> A summary comparing the two logs:
>
> Minimal wallclock time (s) over 10 iterations
>                           Control -> v2                                 Control -> v2+adhocloc+at+every+token
> kdecore.cc -g -O0          10.306548 -> 10.268712: 1.00x faster          10.247160 -> 10.444528: 1.02x slower
> kdecore.cc -g -O1          27.026285 -> 27.220654: 1.01x slower          27.280681 -> 27.622676: 1.01x slower
> kdecore.cc -g -O2          43.791668 -> 44.020270: 1.01x slower          43.904934 -> 44.248477: 1.01x slower
> kdecore.cc -g -O3          47.471836 -> 47.651101: 1.00x slower          47.645985 -> 48.005495: 1.01x slower
> kdecore.cc -g -Os          31.678652 -> 31.802829: 1.00x slower          31.741484 -> 32.033478: 1.01x slower
>    empty.c -g -O0            0.012662 -> 0.011932: 1.06x faster            0.012888 -> 0.013143: 1.02x slower
>    empty.c -g -O1            0.012685 -> 0.012558: 1.01x faster            0.013164 -> 0.012790: 1.03x faster
>    empty.c -g -O2            0.012694 -> 0.012846: 1.01x slower            0.012912 -> 0.013175: 1.02x slower
>    empty.c -g -O3            0.012654 -> 0.012699: 1.00x slower            0.012596 -> 0.012792: 1.02x slower
>    empty.c -g -Os            0.013057 -> 0.012766: 1.02x faster            0.012691 -> 0.012885: 1.02x slower
> big-code.c -g -O0            3.292680 -> 3.325748: 1.01x slower            3.292948 -> 3.303049: 1.00x slower
> big-code.c -g -O1          15.701810 -> 15.765014: 1.00x slower          15.714116 -> 15.759254: 1.00x slower
> big-code.c -g -O2          22.575615 -> 22.620187: 1.00x slower          22.567406 -> 22.605435: 1.00x slower
> big-code.c -g -O3          52.423586 -> 52.590075: 1.00x slower          52.421460 -> 52.703835: 1.01x slower
> big-code.c -g -Os          21.153980 -> 21.253598: 1.00x slower          21.146266 -> 21.260138: 1.01x slower
> influence.i -g -O0            0.148229 -> 0.149518: 1.01x slower            0.148672 -> 0.156262: 1.05x slower
> influence.i -g -O1            0.387397 -> 0.389930: 1.01x slower            0.387734 -> 0.396655: 1.02x slower
> influence.i -g -O2            0.587514 -> 0.589604: 1.00x slower            0.588064 -> 0.596510: 1.01x slower
> influence.i -g -O3            1.273561 -> 1.280514: 1.01x slower            1.274599 -> 1.287596: 1.01x slower
> influence.i -g -Os            0.526045 -> 0.527579: 1.00x slower            0.526827 -> 0.535635: 1.02x slower
>
>
> Maximal ggc memory (kb)
>                      Control -> v2                                 Control -> v2+adhocloc+at+every+token
> kdecore.cc -g -O0      650337.000 -> 654435.000: 1.0063x larger      650337.000 -> 711775.000: 1.0945x larger
> kdecore.cc -g -O1      931966.000 -> 940144.000: 1.0088x larger      931951.000 -> 989384.000: 1.0616x larger
> kdecore.cc -g -O2    1125325.000 -> 1133514.000: 1.0073x larger    1125318.000 -> 1182384.000: 1.0507x larger
> kdecore.cc -g -O3    1221408.000 -> 1229596.000: 1.0067x larger    1221410.000 -> 1278658.000: 1.0469x larger
> kdecore.cc -g -Os      867140.000 -> 871235.000: 1.0047x larger      867141.000 -> 928700.000: 1.0710x larger
>    empty.c -g -O0          1189.000 -> 1192.000: 1.0025x larger          1189.000 -> 1193.000: 1.0034x larger
>    empty.c -g -O1          1189.000 -> 1192.000: 1.0025x larger          1189.000 -> 1193.000: 1.0034x larger
>    empty.c -g -O2          1189.000 -> 1192.000: 1.0025x larger          1189.000 -> 1193.000: 1.0034x larger
>    empty.c -g -O3          1189.000 -> 1192.000: 1.0025x larger          1189.000 -> 1193.000: 1.0034x larger
>    empty.c -g -Os          1189.000 -> 1192.000: 1.0025x larger          1189.000 -> 1193.000: 1.0034x larger
> big-code.c -g -O0      166584.000 -> 172731.000: 1.0369x larger      166584.000 -> 172726.000: 1.0369x larger
> big-code.c -g -O1      279793.000 -> 285940.000: 1.0220x larger      279793.000 -> 285935.000: 1.0220x larger
> big-code.c -g -O2      400058.000 -> 406194.000: 1.0153x larger      400058.000 -> 406189.000: 1.0153x larger
> big-code.c -g -O3      903648.000 -> 909750.000: 1.0068x larger      903906.000 -> 910001.000: 1.0067x larger
> big-code.c -g -Os      357060.000 -> 363010.000: 1.0167x larger      357060.000 -> 363005.000: 1.0166x larger
> influence.i -g -O0          9273.000 -> 9719.000: 1.0481x larger         9273.000 -> 13303.000: 1.4346x larger
> influence.i -g -O1        12968.000 -> 13414.000: 1.0344x larger        12968.000 -> 16998.000: 1.3108x larger
> influence.i -g -O2        16386.000 -> 16768.000: 1.0233x larger        16386.000 -> 20352.000: 1.2420x larger
> influence.i -g -O3        35508.000 -> 35763.000: 1.0072x larger        35508.000 -> 39346.000: 1.1081x larger
> influence.i -g -Os        14287.000 -> 14669.000: 1.0267x larger        14287.000 -> 18253.000: 1.2776x larger
>
> Thoughts?

The compile-time and memory-usage impact for the adhocloc at every
token patchkit is quite big.  Remember
that gaining 1% in compile-time is hard and 20-40% memory increase for
influence.i looks too much.

I also wonder why you see differences in memory usage change for
different -O levels.  I think we should
have a pretty "static" line table after parsing?  Thus rather than
percentages I'd like to see absolute changes
(which I'd expect to be the same for all -O levels).

Richard.

> Dave
>
>