[PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

[Martin Liška]

@Richi: Can you please reply to this email?

On 5/21/21 10:43 AM, Martin Liška wrote:
> On 5/20/21 2:54 PM, Richard Biener wrote:
>> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote:
>>>
>>> Hello.
>>>
>>> I've got a patch candidate that leverages partial linking for a couple of selected object files.
>>>
>>> I'm sending make all-host- jX results for my machine:
>>>
>>> before: 3m18s (user 32m52s)
>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg
>>>
>>> after: 2m57m (user 35m)
>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg
>>>
>>> One can utilize it with:
>>> make -j16 all-host PARTIAL_LTO=1
>>>
>>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap?
>>> One can also tweak --param=lto-partitions=16 param value.
>>>
>>> Thoughts?
>>
>> You're LTO linking multiple objects here - that's almost as if you
>> were doing this
>> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
>> libbackend.a rule do a similar partial link trick.
> 
> Yeah, apart from that one can't likely do partial linking for an archive:
> 
> $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a
> collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
> compilation terminated.
> 
> while ld.bfd immediately finishes.
> 
>>
>> That gets you half of a LTO bootstrap then.
>>
>> So why did you go from applying this per-file to multiple files?  Does $(LINKER)
>> have a proper rule to pick up a jobserver?
>>
>> When upstreaming in any form you probably have to gate it on bootstrap-lto
>> being not active.
> 
> Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right?
> 
> One more thing I face is broken dependency:
> $ make clean && make -j32 PARTIAL_LTO=1
> 
> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto
> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto
> 
> In file included from ./tm.h:26,
>                   from /home/marxin/Programming/gcc/gcc/backend.h:28,
>                   from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23,
>                   from generic-match.c:4:
> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
>   2286 | #include "insn-attr-common.h"
>        |          ^~~~~~~~~~~~~~~~~~~~
> compilation terminated.
> make: *** [Makefile:2678: generic-match-lto.o] Error 1
> make: *** Waiting for unfinished jobs....
> 
> In file included from ./tm.h:26,
>                   from /home/marxin/Programming/gcc/gcc/backend.h:28,
>                   from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23,
>                   from gimple-match.c:4:
> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
>   2286 | #include "insn-attr-common.h"
>        |          ^~~~~~~~~~~~~~~~~~~~
> 
> I explicitly added:
> gimple-match.o: gimple-match.c $(generated_files)
> generic-match.o: generic-match.c $(generated_files)
> 
> But it's not obeyed.
> 
> Martin
> 
>>
>> Richard.
>>
>>> Thanks,
>>> Martin
>