[PATCH] Add .gnu.lto_.meta section.

[Jan Hubicka]

> On 6/21/19 2:34 PM, Richard Biener wrote:
> > On Fri, Jun 21, 2019 at 12:20 PM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> Hi.
> >>
> >> The patch is about a new ELF section that will contain information
> >> about LTO version. And for the future, used compression will be stored
> >> here. The patch removes streaming of the version into each section.
> > 
> > I'd like each section to have a header containing a compression method
> > (compressed or not, and now zlib vs. zstd).  We currently have a mix
> > and knowledge is hidden.
> 
> That would be possible, good idea.
> 
> > 
> > I also remember I had old patches to make the data streamer compress
> > the stream on-the-fly, not requiring the full uncompressed stream in
> > memory and then re-access it for compression (needing more temporary
> > memory).  But IIRC the speedup was marginal.

It may be more interesting as memory optimization now we do parallel WPA
streaming BTW.
> > 
> > My thought is also that we should have exactly _one_ ELF section
> > for the LTO bytecode and our own container inside (much like
> > simple-object does for non-ELF).  So having another one is, well, ugly ;)
> 
> Having N sections for all symbols (functions and variables) is handy because
> we read some of the during WPA (by IPA ICF). So having all in a single section
> would make decompression more complicated.

There is nothing preventing us from implementing our own subsections in
that section where each is compressed independently and can be copied to
output file and still save size of headers needed for that.

For example all sections are keyed by the lto section enum + possibly a
decl that can be both streamed as integers rather than quite lenghtly
strings.  (the decl to name is streamed into the global decl stream and
available at all time).

I guess one can go for one global LTO major/minor in the section header
and then  section type/is compressed/size + optional decl ID for each
subsection reducing headers to about 1+8+optional 8 bytes per header.
This can be all placed at the begining of the file followed by RAW data
of individual sections.  I bet this is a lot smaller than ELF overhead :)
> 
> I'm going to prepare a patch that will pull out a LTO section header
> from compressed stream.
This looks like good step (and please stream it in host independent
way). I suppose all these issues can be done one-by-one.

Honza
> 
> Martin
> 
> > 
> >> Disadvantage is a format change that will lead to following errors when
> >> LTO bytecode is used from a different version:
> >>
> >> $ gcc x.o
> >> lto1: fatal error: bytecode stream in file ‘x.o’ generated with GCC compiler older than 10.0
> >>
> >> $ gcc-9 main.o
> >> lto1: fatal error: bytecode stream in file ‘main.o’ generated with LTO version 850.0 instead of the expected 8.0
> >>
> >> I've been testing the patch.
> >> Thoughts?
> >> Martin
>