[RFC] Getting LTO incremental linking work

[H.J. Lu]

On Wed, Nov 25, 2015 at 3:15 AM, Richard Biener <rguenther@suse.de> wrote:
> On Wed, 25 Nov 2015, Jan Hubicka wrote:
>
>> Hi,
>> PR 67548 is about LTO not supporting incremental linking.  I never really
>> considered our current incremental linking very useful, because it triggers
>> code generation at the incremental link time basically nullifying any
>> benefits of whole program optimization and in fact I think it is harmful,
>> because it sort of works and w/o any warning produce not very optimized code.
>>
>> Basically there are 3 schemes how to make incremental link work
>>  1) Turn LTO objects to non-LTO as we do now
>>  2) concatenate LTO sections as implemented by Andi and Hj
>>  3) Do actual linking of LTO sections
>>
>> The problem of current implementation of 1) is that GCC thinks the resulting
>> object file will not be used for static linking and thus assume that hidden
>> symbols can be turned to static.
>>
>> In the log of PR67548 HJ actually pointed out that we do have API at linker
>> plugin side which says what type of output is done.  This is cool because we
>> can also use it to drop -fpic when building static binary. This is common in
>> Firefox, where some objects are built with -fpic and linked to both binaries
>> and libraries.
>>
>> Moreover we do have all infrastructure ready to implement 3).  Our tree merging
>> and symbol table handling is fuly incremental and I think made a patch to
>> implement it today.   The scheme is easy:
>>
>>  1) linker plugin is modified to pass -flinker-output to lto wrapper
>>     linker-output is either dyn (.so), pie or exec
>>     for incremental linking I added .rel for 3) and noltorel for 1)
>>
>>     currently it does rel because 3) (nor 2) can not be done when incremnetal
>>     linking is done on both LTO and non-LTO objects.
>
> That's because the result would be a "fat" object where both pieces
> would be needed.  Btw, I wonder why you are not running into the
> same issues as me when producing linker plugin output (the "merged"
> LTO IL) that is LTO IL.  Ah, possibly because the link is incremental,
> and thus all special-handling of LTO sections is disabled.
>
>>     In this case linker
>>     plugin output warings about code quality loss and switch to
>>     noltorel.
>>  2) with -flinker-ouptut the lto wrapper behaves same way as with
>>     -flto-partition=none.
>>  3) lto frontend parses -flinker-output and sets our internal flags accordingly.
>>     I added new flag_incremental_linking to inform middle-end about the fact
>>     that the output is going to be statically linked again.  This disables
>>     the privatization of hidden symbols and if set to 2 it also triggers
>>     the LTO IL streaming
>
> I wonder why it behaves like -flto-partition=none in the case it does
> not need to do LTO IL streaming (which I hope does LTO IL streaming
> only?  or does this implement fat objects "correctly"?).  Can't
> we still parallelize the build via LTRANS and then incrementally
> link the result (I suppose the linker will do that for us with the
> linker plugin outputs already?)?
>
> -flto-partition=none itself isn't more memory intensive than
> WPA in these days, it's only about compile-time, correct?
>
> Your patch means that Andis/HJs work is no longer needed and we can
> drop the section suffixes again?
>
>

There is a difference between "ld -r " and "gcc -r". "ld -r" may not
perform any LTO.

-- 
H.J.