Re: [PATCH] Enable parallel ltrans stage with -fwhopr=N

[Richard Guenther <rguenther at suse dot de>]

On Tue, 25 May 2010, Ralf Wildenhues wrote:

> * Richard Guenther wrote on Tue, May 25, 2010 at 03:03:23PM CEST:
> > On Sun, 23 May 2010, Richard Guenther wrote:
> > > On Sun, May 23, 2010 at 9:16 AM, Ralf Wildenhues wrote:
> > > > I further noted that when interrupted with C^c, -fwhopr= may leave a
> > > > number of files matching *.wpa.o and *.wpa.ltrans.o around in the build
> > > > tree (not sure if only the current directory or also subdirs). ?Besides
> > > > the missing cleanup issue (interrupted make only removes incompletely
> > > > updated targets), are these files suitably named so that they won't
> > > > interfere with, say, toplevel parallel make generating prog1 and prog2
> > > > from the same (or an overlapping) set of *.o files but with possibly
> > > > different link flags? ?These names don't look random to me.
> > > >
> > > > I get some leftover files even if 'make' for some reason doesn't run at
> > > > all or doesn't comprehend -j.
> > > 
> > > Yeah, that's easily fixed - I can look at it.
> > 
> > I have posted a patch to fix that.
> 
> What exactly is "that"?  The patch you posted does not fix all the
> issues I described in the above two paragraphs.  There are still
> leftover *.wpa.o files after interrupt of failed 'make', and the file
> names still are not parallel-safe.  (That doesn't mean the patch is
> bad, nor that I expect you to fix the other issues.) Should I open a PR?

Hm, that would be a bug (the leftover *.wpa.o files).  It should
detect a failed make in collect_execute and end up calling 
lto_wrapper_exit which in turn should unlink those files.

The file names should be parallel-safe - or what are you refering
to here?

> > What would be nice is to automatically do the LTRANS phase via
> > $(MAKE) if we know a jobserver is running (basically execute
> > like a regular sub-make).  Can we somehow detect that easily?
> 
> Do you mean you would like to connect to the jobserver from a calling
> make process?  That works only if the higher-level make opened job
> server file descriptors for you, which GNU make only does if the rule
> command to be invoked contained the strings ${MAKE} or $(MAKE) in the
> makefile, or was preceded by a '+' sign.  Neither of which is typical
> for link commands in makefiles, nor would it be advisable, because it
> would let 'make -n' actually try to execute the commands, too.
> 
> This might be another argument to convince GNU make to allow separating
> the notion of make-recursion from the notion of allowing to pass job
> server file descriptors.  I will take this up with bug-make.
> 
> 
> If you didn't mean that, then I haven't understood the question.  You

Yes, that was what I meant.

> can typically detect by looking at MAKEFLAGS whether parallel GNU make
> or a parallel non-GNU make is running somewhere up the chain, but you
> can't communicate with them.  Enabling parallelism merely because it is
> done at a higher level, too, can be a dangerous business, especially
> since whopr can use large amounts of memory, and compiling or linking
> several large pieces of code typically does not allow the system to
> share a lot of memory.  E.g., with a medium-sized C++ project over here,
> 'make -j' or -jN with large N is effectively a slow short-hand for
> reboot.

Right, so let's leave things as they are for now.

> > If not we might want to still make that the default for -fwhopr
> > and allow -fwhopr=0 to force it back to direct invocation of
> > LTRANS processes.
> > 
> > Would it be as simple (with the last interface change) as
> > invocating $(MAKE) if set?
> 
> Hope my reply above addresses these questions, too.

Yes, thanks.
Richard.