[Bug lto/65536] LTO line number information garbled

[hubicka at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65536

--- Comment #53 from Jan Hubicka <hubicka at gcc dot gnu.org> ---
> You can get an estimate of how much memory would be required to stream in/out 
> directly the line_table by summing up the memory reported by 
> dump_line_table_statistics for each TU before streaming out (perhaps using 
> -ftrack-macro-expansion=0 to reduce it further). This would still be an 
> overestimate, because one can drop one char (sysp) and one int (included_from) > per map and one can drop all maps that are not used by LTO. Moreover, you will > not need a cache and everything will be in order already when you stream in.

I see, the stats actually are in -fmem-report, how convenient ;))

With Firefox I get:
Line Table allocations during the compilation process
Number of ordinary maps used:          407k
Ordinary map used size:                 15M
Number of ordinary maps allocated:     409k
Ordinary maps allocated size:           15M
Number of macro maps used:               0 
Macro maps used size:                    0 
Macro maps locations size:               0 
Macro maps size:                         0 
Duplicated maps locations size:          0 
Total allocated maps size:              15M
Total used maps size:                   15M

after streaming in all declarations&types and

Line Table allocations during the compilation process
Number of ordinary maps used:          769k
Ordinary map used size:                 30M
Number of ordinary maps allocated:    1638k
Ordinary maps allocated size:           63M
Number of macro maps used:               0 
Macro maps used size:                    0 
Macro maps locations size:               0 
Macro maps size:                         0 
Duplicated maps locations size:          0 
Total allocated maps size:              63M
Total used maps size:                   30M

by end of WPA stage. The extra growth is by variable initializers and function
bodies we bring in on demand (for merging, ICF, devirt machinery). Those by
large part bypass my cache.

Individual ltrans units consume after streaming in global decls:
Line Table allocations during the compilation process
Number of ordinary maps used:           29k
Ordinary map used size:               1189k
Number of ordinary maps allocated:     102k
Ordinary maps allocated size:         4095k
Number of macro maps used:               0 
Macro maps used size:                    0 
Macro maps locations size:               0 
Macro maps size:                         0 
Duplicated maps locations size:          0 
Total allocated maps size:            4095k
Total used maps size:                 1189k

this does not look terrible

I tried to sum the sizes for EON, and got to 92493kB and 8631 without macro
expansion tracking (note that this size is basically the same with or without
my proposed line-maps.c patch, so it does not seem to pesimize non-LTO builds).

At WPA time (with my proposed patch, will rebuild tree without after lunch) I
get:

Line Table allocations during the compilation process
Number of ordinary maps used:         6084 
Ordinary map used size:                237k
Number of ordinary maps allocated:    6553 
Ordinary maps allocated size:          255k
Number of macro maps used:               0 
Macro maps used size:                    0 
Macro maps locations size:               0 
Macro maps size:                         0 
Duplicated maps locations size:          0 
Total allocated maps size:             255k
Total used maps size:                  237k

So about 30x smaller, even though we do have information loss here (i.e. inline
stacks)

So it seems we can not really directly stream linemaps - there is way too much
of information that is discarded from parsing time to LTO time.
(about 95% of trees that are streamed out are discarded by tree merging and
thus with my cache never hits linemaps).

Linemap representation is interesting. I suppose for next release we want to do
kind of on-disk variant inspired by libcpp.  If we arrange cache to be
per-LTO-section (not randomly flushed as it is now), we can read the on-disk
format into LTO own format and apply relevant part to the line-maps after tree
merging.