Re: [PATCH] Enable parallel ltrans stage with -fwhopr=N

[Ralf Wildenhues <Ralf dot Wildenhues at gmx dot de>]

* Richard Guenther wrote on Tue, May 25, 2010 at 03:03:23PM CEST:
> On Sun, 23 May 2010, Richard Guenther wrote:
> > On Sun, May 23, 2010 at 9:16 AM, Ralf Wildenhues wrote:
> > > I further noted that when interrupted with C^c, -fwhopr= may leave a
> > > number of files matching *.wpa.o and *.wpa.ltrans.o around in the build
> > > tree (not sure if only the current directory or also subdirs). ?Besides
> > > the missing cleanup issue (interrupted make only removes incompletely
> > > updated targets), are these files suitably named so that they won't
> > > interfere with, say, toplevel parallel make generating prog1 and prog2
> > > from the same (or an overlapping) set of *.o files but with possibly
> > > different link flags? ?These names don't look random to me.
> > >
> > > I get some leftover files even if 'make' for some reason doesn't run at
> > > all or doesn't comprehend -j.
> > 
> > Yeah, that's easily fixed - I can look at it.
> 
> I have posted a patch to fix that.

What exactly is "that"?  The patch you posted does not fix all the
issues I described in the above two paragraphs.  There are still
leftover *.wpa.o files after interrupt of failed 'make', and the file
names still are not parallel-safe.  (That doesn't mean the patch is
bad, nor that I expect you to fix the other issues.) Should I open a PR?

> What would be nice is to automatically do the LTRANS phase via
> $(MAKE) if we know a jobserver is running (basically execute
> like a regular sub-make).  Can we somehow detect that easily?

Do you mean you would like to connect to the jobserver from a calling
make process?  That works only if the higher-level make opened job
server file descriptors for you, which GNU make only does if the rule
command to be invoked contained the strings ${MAKE} or $(MAKE) in the
makefile, or was preceded by a '+' sign.  Neither of which is typical
for link commands in makefiles, nor would it be advisable, because it
would let 'make -n' actually try to execute the commands, too.

This might be another argument to convince GNU make to allow separating
the notion of make-recursion from the notion of allowing to pass job
server file descriptors.  I will take this up with bug-make.


If you didn't mean that, then I haven't understood the question.  You
can typically detect by looking at MAKEFLAGS whether parallel GNU make
or a parallel non-GNU make is running somewhere up the chain, but you
can't communicate with them.  Enabling parallelism merely because it is
done at a higher level, too, can be a dangerous business, especially
since whopr can use large amounts of memory, and compiling or linking
several large pieces of code typically does not allow the system to
share a lot of memory.  E.g., with a medium-sized C++ project over here,
'make -j' or -jN with large N is effectively a slow short-hand for
reboot.

> If not we might want to still make that the default for -fwhopr
> and allow -fwhopr=0 to force it back to direct invocation of
> LTRANS processes.
> 
> Would it be as simple (with the last interface change) as
> invocating $(MAKE) if set?

Hope my reply above addresses these questions, too.

Cheers,
Ralf