[PATCH] Try LTO partial linking. (Was: Speed of compiling gimple-match.c)

Tue Jun 1 13:19:27 GMT 2021

On Tue, Jun 1, 2021 at 1:25 PM Martin Liška <mliska@suse.cz> wrote:
>
> On 6/1/21 9:42 AM, Richard Biener wrote:
> > On Tue, Jun 1, 2021 at 9:33 AM Martin Liška <mliska@suse.cz> wrote:
> >>
> >> @Richi: Can you please reply to this email?
> >
> > Not sure what I should add here?  Honza suggested to mangle the
> > promoted symbol names.
>
> Sure and I sent a patch for that.
>
> > I don't
> > really like the idea to compile multiple TUs into one object.  Also
>
> What's problematic is that we'll have to wait for one another release to make it useful
> (if you don't want to build the current master with a snapshot compiler).

IMHO it's a bugfix.  Note that I'm not sure what the intent of the change is.
If it is to speedup bootstrap then using LTO bootstrap would do the trick
as well (and better) if we'd simply process all of libbackend.a this way
(and thus avoid re-linking that once for each frontend).  If it is to speedup
dev (re-)builds then dragging in more files will make it build longer since
for example insn-recog.c may be unchanged but gimple-match.c not.

> > +LTO_LINKER_FLAGS = -flto=auto --param=lto-partitions=16
> > -flinker-output=nolto-rel -r
> >
> > why hard-code to 16 partitions?  You're side-stepping the driver
> > diagnostic by doing
> > compile & link separately, but in the end we're going to want sth like Giulianos
> > -fparallel-compile that works transparently from within the driver, so
> > the "manual"
> > operation should try to follow that or alternatively a driver-only
> > wrapper around the
> > "manual" processing could be added whose implementation can be optimized later.
>
> All right. Do you want me refreshing his -fparallel-compile option introduction?

I'm not sure if we've arrived at mergeable state - but if it's
reasonably possible
to hide s/-fparallel-compile/-flto -r -flinker-output=nolto-rel/ split
into compile & link
parts (avoiding the diagnostic on -flinker-output) in the driver then
I think that's
a very reasonable first step (after fixing the symbol privatization issue).  The
GSOC project then was to elide the IL streaming from the high-level operation.

Richard,

> >
> > Why do you use -flto=auto?  There should be a jobserver active.
>
> Yes, that should not be needed.
>
> Martin
>
> >
> >> On 5/21/21 10:43 AM, Martin Liška wrote:
> >>> On 5/20/21 2:54 PM, Richard Biener wrote:
> >>>> On Thu, May 20, 2021 at 2:34 PM Martin Liška <mliska@suse.cz> wrote:
> >>>>>
> >>>>> Hello.
> >>>>>
> >>>>> I've got a patch candidate that leverages partial linking for a couple of selected object files.
> >>>>>
> >>>>> I'm sending make all-host- jX results for my machine:
> >>>>>
> >>>>> before: 3m18s (user 32m52s)
> >>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/1dd5eae5001295ba0230a689f7edc67284c9b742/gcc-all-host.svg
> >>>>>
> >>>>> after: 2m57m (user 35m)
> >>>>> https://gist.githubusercontent.com/marxin/223890df4d8d8e490b6b2918b77dacad/raw/d659b2187cf622167841efbbe6bc93cb33855fa9/gcc-all-host-partial-lto.svg
> >>>>>
> >>>>> One can utilize it with:
> >>>>> make -j16 all-host PARTIAL_LTO=1
> >>>>>
> >>>>> @Segher, Andrew: Can you please measure time improvement for your slow bootstrap?
> >>>>> One can also tweak --param=lto-partitions=16 param value.
> >>>>>
> >>>>> Thoughts?
> >>>>
> >>>> You're LTO linking multiple objects here - that's almost as if you
> >>>> were doing this
> >>>> for the whole of libbackend.a ... so $(OBJS)_CLFAGS += -flto and in the
> >>>> libbackend.a rule do a similar partial link trick.
> >>>
> >>> Yeah, apart from that one can't likely do partial linking for an archive:
> >>>
> >>> $ g++ -no-pie -flto=auto --param=lto-partitions=16 -flinker-output=nolto-rel -r libbackend.a
> >>> collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
> >>> compilation terminated.
> >>>
> >>> while ld.bfd immediately finishes.
> >>>
> >>>>
> >>>> That gets you half of a LTO bootstrap then.
> >>>>
> >>>> So why did you go from applying this per-file to multiple files?  Does $(LINKER)
> >>>> have a proper rule to pick up a jobserver?
> >>>>
> >>>> When upstreaming in any form you probably have to gate it on bootstrap-lto
> >>>> being not active.
> >>>
> >>> Sure, that's reasonable, we can likely detect a -flto option in $(COMPILE), right?
> >>>
> >>> One more thing I face is broken dependency:
> >>> $ make clean && make -j32 PARTIAL_LTO=1
> >>>
> >>> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o gimple-match-lto.o -MT gimple-match-lto.o -MMD -MP -MF ./.deps/gimple-match-lto.TPo gimple-match.c -flto
> >>> g++ -fcf-protection -fno-PIE -c   -g   -DIN_GCC -fPIC    -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wno-error=format-diag -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -fno-common -Wno-unused -DHAVE_CONFIG_H -I. -I. -I/home/marxin/Programming/gcc/gcc -I/home/marxin/Programming/gcc/gcc/. -I/home/marxin/Programming/gcc/gcc/../include -I/home/marxin/Programming/gcc/gcc/../libcpp/include -I/home/marxin/Programming/gcc/gcc/../libcody  -I/home/marxin/Programming/gcc/gcc/../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libdecnumber/bid -I../libdecnumber -I/home/marxin/Programming/gcc/gcc/../libbacktrace   -o generic-match-lto.o -MT generic-match-lto.o -MMD -MP -MF ./.deps/generic-match-lto.TPo generic-match.c -flto
> >>>
> >>> In file included from ./tm.h:26,
> >>>                    from /home/marxin/Programming/gcc/gcc/backend.h:28,
> >>>                    from /home/marxin/Programming/gcc/gcc/generic-match-head.c:23,
> >>>                    from generic-match.c:4:
> >>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
> >>>    2286 | #include "insn-attr-common.h"
> >>>         |          ^~~~~~~~~~~~~~~~~~~~
> >>> compilation terminated.
> >>> make: *** [Makefile:2678: generic-match-lto.o] Error 1
> >>> make: *** Waiting for unfinished jobs....
> >>>
> >>> In file included from ./tm.h:26,
> >>>                    from /home/marxin/Programming/gcc/gcc/backend.h:28,
> >>>                    from /home/marxin/Programming/gcc/gcc/gimple-match-head.c:23,
> >>>                    from gimple-match.c:4:
> >>> /home/marxin/Programming/gcc/gcc/config/i386/i386.h:2286:10: fatal error: insn-attr-common.h: No such file or directory
> >>>    2286 | #include "insn-attr-common.h"
> >>>         |          ^~~~~~~~~~~~~~~~~~~~
> >>>
> >>> I explicitly added:
> >>> gimple-match.o: gimple-match.c $(generated_files)
> >>> generic-match.o: generic-match.c $(generated_files)
> >>>
> >>> But it's not obeyed.
> >>>
> >>> Martin
> >>>
> >>>>
> >>>> Richard.
> >>>>
> >>>>> Thanks,
> >>>>> Martin
> >>>
> >>
>