[PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

Martin Liška mliska@suse.cz
Tue Jun 1 11:25:56 GMT 2021


On 6/1/21 9:42 AM, Richard Biener wrote:
> On Tue, Jun 1, 2021 at 9:33 AM Martin Liška <mliska@suse.cz> wrote:
>>
>> @Richi: Can you please reply to this email?
> 
> Not sure what I should add here?  Honza suggested to mangle the
> promoted symbol names.

Sure and I sent a patch for that.

> I don't
> really like the idea to compile multiple TUs into one object.  Also

What's problematic is that we'll have to wait for one another release to make it useful
(if you don't want to build the current master with a snapshot compiler).

> 
> +LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16
> -flinker-output=nolto-rel -r
> 
> why hard-code to 16 partitions?  You're side-stepping the driver
> diagnostic by doing
> compile & link separately, but in the end we're going to want sth like Giulianos
> -fparallel-compile that works transparently from within the driver, so
> the "manual"
> operation should try to follow that or alternatively a driver-only
> wrapper around the
> "manual" processing could be added whose implementation can be optimized later.

All right. Do you want me refreshing his -fparallel-compile option introduction?

> 
> Why do you use -flto=auto?  There should be a jobserver active.

Yes, that should not be needed.

Martin

> 
>> On 5/21/21 10:43 AM, Martin Liška wrote:
>>> On 5/20/21 2:54 PM, Richard Biener wrote:
>>>> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote:
>>>>>
>>>>> Hello.
>>>>>
>>>>> I've got a patch candidate that leverages partial linking for a couple of selected object files.
>>>>>
>>>>> I'm sending make all-host- jX results for my machine:
>>>>>
>>>>> before: 3m18s (user 32m52s)
>>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg
>>>>>
>>>>> after: 2m57m (user 35m)
>>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg
>>>>>
>>>>> One can utilize it with:
>>>>> make -j16 all-host PARTIAL_LTO=1
>>>>>
>>>>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap?
>>>>> One can also tweak --param=lto-partitions=16 param value.
>>>>>
>>>>> Thoughts?
>>>>
>>>> You're LTO linking multiple objects here - that's almost as if you
>>>> were doing this
>>>> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
>>>> libbackend.a rule do a similar partial link trick.
>>>
>>> Yeah, apart from that one can't likely do partial linking for an archive:
>>>
>>> $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a
>>> collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
>>> compilation terminated.
>>>
>>> while ld.bfd immediately finishes.
>>>
>>>>
>>>> That gets you half of a LTO bootstrap then.
>>>>
>>>> So why did you go from applying this per-file to multiple files?  Does $(LINKER)
>>>> have a proper rule to pick up a jobserver?
>>>>
>>>> When upstreaming in any form you probably have to gate it on bootstrap-lto
>>>> being not active.
>>>
>>> Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right?
>>>
>>> One more thing I face is broken dependency:
>>> $ make clean && make -j32 PARTIAL_LTO=1
>>>
>>> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto
>>> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto
>>>
>>> In file included from ./tm.h:26,
>>>                    from /home/marxin/Programming/gcc/gcc/backend.h:28,
>>>                    from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23,
>>>                    from generic-match.c:4:
>>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
>>>    2286 | #include "insn-attr-common.h"
>>>         |          ^~~~~~~~~~~~~~~~~~~~
>>> compilation terminated.
>>> make: *** [Makefile:2678: generic-match-lto.o] Error 1
>>> make: *** Waiting for unfinished jobs....
>>>
>>> In file included from ./tm.h:26,
>>>                    from /home/marxin/Programming/gcc/gcc/backend.h:28,
>>>                    from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23,
>>>                    from gimple-match.c:4:
>>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
>>>    2286 | #include "insn-attr-common.h"
>>>         |          ^~~~~~~~~~~~~~~~~~~~
>>>
>>> I explicitly added:
>>> gimple-match.o: gimple-match.c $(generated_files)
>>> generic-match.o: generic-match.c $(generated_files)
>>>
>>> But it's not obeyed.
>>>
>>> Martin
>>>
>>>>
>>>> Richard.
>>>>
>>>>> Thanks,
>>>>> Martin
>>>
>>



More information about the Gcc-patches mailing list