Re: [PATCH] Enable parallel ltrans stage with -fwhopr=N

[Ralf Wildenhues <Ralf dot Wildenhues at gmx dot de>]

* Richard Guenther wrote on Thu, May 27, 2010 at 11:12:19AM CEST:
> On Wed, 26 May 2010, Ralf Wildenhues wrote:
> > * Richard Guenther wrote on Wed, May 26, 2010 at 10:50:24AM CEST:
> > > On Tue, 25 May 2010, Ralf Wildenhues wrote:
> > > > * Richard Guenther wrote on Tue, May 25, 2010 at 03:03:23PM CEST:
> > > > > On Sun, 23 May 2010, Richard Guenther wrote:
> > > > > > On Sun, May 23, 2010 at 9:16 AM, Ralf Wildenhues wrote:
> > > > > > > I further noted that when interrupted with C^c, -fwhopr= may leave a
> > > > > > > number of files matching *.wpa.o and *.wpa.ltrans.o around in the build
> > > > > > > tree (not sure if only the current directory or also subdirs). ?Besides
> > > > > > > the missing cleanup issue (interrupted make only removes incompletely
> > > > > > > updated targets), are these files suitably named so that they won't
> > > > > > > interfere with, say, toplevel parallel make generating prog1 and prog2
> > > > > > > from the same (or an overlapping) set of *.o files but with possibly
> > > > > > > different link flags? ?These names don't look random to me.
> > > > > > >
> > > > > > > I get some leftover files even if 'make' for some reason doesn't run at
> > > > > > > all or doesn't comprehend -j.
> > > > > > 
> > > > > > Yeah, that's easily fixed - I can look at it.
> > > > > 
> > > > > I have posted a patch to fix that.
> > > > 
> > > > What exactly is "that"?  The patch you posted does not fix all the
> > > > issues I described in the above two paragraphs.  There are still
> > > > leftover *.wpa.o files after interrupt of failed 'make', and the file
> > > > names still are not parallel-safe.  (That doesn't mean the patch is
> > > > bad, nor that I expect you to fix the other issues.) Should I open a PR?
> > > 
> > > Hm, that would be a bug (the leftover *.wpa.o files).  It should
> > > detect a failed make in collect_execute and end up calling 
> > > lto_wrapper_exit which in turn should unlink those files.
> > 
> > Except that it doesn't work right yet with r159853.  I can still
> > easily get leftover *.wpa.o as well as *.wpa.ltrans.o files with C^c.
> > Would you like me to create a reproducer?
> 
> Yes, that would be nice.

Well, the easiest thing that I can come up is: in an up to date GCC
build tree,

  cd gcc
  rm -f *.o
  make CFLAGS='-g -O2 -fwhopr=3' xgcc cc1
  rm -f xgcc cc1

Now, just interrupt the link after a couple of seconds (don't do this in
a shell where %1 is another job):

  make CFLAGS='-g -O2 -fwhopr=3' xgcc cc1 & sleep 2; kill -INT %1

You will see a number of leftover files, now with names matching
*.ltrans$N.o and *.ltrans$N.ltrans.o in $TMPDIR.  These files are not
removed upon SIGINT.

You might decide to declare this intentional, but with users having
their twitchy finger near C^c, it can quickly fill /tmp.

> > I can easily provoke failures such as:
> > 
> >   lto-wrapper: deleting LTRANS file a_b_c.wpa.o: No such file or directory
> >   lto-wrapper: deleting LTRANS file /tmp/cccUSDhq.args: No such file or directory
> >   collect2: lto-wrapper returned 1 exit status
> > 
> > with a real-world setup similar to above.  The second error message looks
> > like it could be exploitable, by the way.
> 
> Oh, indeed.  I see what you are refering to now.   Note that the
> names should be all tmpnames if -save-temps is not specified.

That seems to be the case now, and looks good to me.

> I'm not sure if we care enough for the -save-temps case to be
> parallel-safe - we definitely want to have user-recognizable file-names
> here.

I agree.

> The above messages could also stem from us trying to delete files that
> have been deleted by someone else or have not yet been created
> (only at failure time).  Those should be fixed by

I cannot reproduce the messages any more.

Cheers,
Ralf