Bug 48086 - bootstrap-lto creates c-common.s with too many sections on x86_64-apple-darwin10
bootstrap-lto creates c-common.s with too many sections on x86_64-apple-darwin10
Status: RESOLVED FIXED
Product: gcc
Classification: Unclassified
Component: lto
4.6.0
: P3 normal
: ---
Assigned To: Not yet assigned to anyone
:
: 49228 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-03-12 02:15 UTC by Jack Howarth
Modified: 2011-05-30 09:11 UTC (History)
7 users (show)

See Also:
Host: *-apple-darwin{9,10}
Target: *-apple-darwin{9,10}
Build: *-apple-darwin{9,10}
Known to work:
Known to fail:
Last reconfirmed: 2011-03-12 09:11:09


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jack Howarth 2011-03-12 02:15:35 UTC
The number of sections in mach-o has always been restricted to 255. The mach-o nlist struct has a one byte n_sect field whichis the index of the section that symbol is in. Previous to Xcode 4.0, the assembler did not validate that the n_sect field was not overflowed but now does. This causes the lto-bootstrap on x86_64-apple-darwin10 to fail with...

/sw/src/fink.build/gcc46-4.6.0-1000/darwin_objdir/./prev-gcc/xgcc -B/sw/src/fink.build/gcc46-4.6.0-1000/darwin_objdir/./prev-gcc/ -B/sw/lib/gcc4.6/x86_64-apple-darwin10.7.0/bin/ -B/sw/lib/gcc4.6/x86_64-apple-darwin10.7.0/bin/ -B/sw/lib/gcc4.6/x86_64-apple-darwin10.7.0/lib/ -isystem /sw/lib/gcc4.6/x86_64-apple-darwin10.7.0/include -isystem /sw/lib/gcc4.6/x86_64-apple-darwin10.7.0/sys-include    -c  -DIN_GCC_FRONTEND -DIN_GCC_FRONTEND -DIN_GCC_FRONTEND -g -O2 -mdynamic-no-pic -flto=jobserver -frandom-seed=1 -DIN_GCC   -W -Wall -Wwrite-strings -Wcast-qual -Wstrict-prototypes -Wmissing-prototypes  -Wold-style-definition -Wc++-compat -fno-common  -DHAVE_CONFIG_H -I. -Ic-family -I../../gcc-4.6-20110311/gcc -I../../gcc-4.6-20110311/gcc/c-family -I../../gcc-4.6-20110311/gcc/../include -I../../gcc-4.6-20110311/gcc/../libcpp/include -I/sw/include -I/sw/include  -I../../gcc-4.6-20110311/gcc/../libdecnumber -I../../gcc-4.6-20110311/gcc/../libdecnumber/dpd -I../libdecnumber -I/sw/include  -I/sw/include -DCLOOG_INT_GMP -DCLOOG_ORG -I/sw/include ../../gcc-4.6-20110311/gcc/c-family/c-common.c -o c-family/c-common.o
/var/tmp//ccYAptak.s:297257:FATAL:too many sections (maximum 255)

make[3]: *** [c-family/c-common.o] Error 1
make[2]: *** [all-stage2-gcc] Error 2
make[1]: *** [stage2-bubble] Error 2
make: *** [all] Error 2

which, with -v, appears as....

/sw/src/fink.build/gcc46-4.6.0-1000/darwin_objdir/./prev-gcc/as -arch x86_64 -force_cpusubtype_ALL -o c-family/c-common.o c-common.s
c-common.s:298717:FATAL:too many sections (maximum 255)
Comment 1 Jack Howarth 2011-03-12 02:16:22 UTC
  $ ../gcc-4.6-20110311/configure --prefix=/sw --prefix=/sw/lib/gcc4.6 --mandir=/sw/share/man --infodir=/sw/lib/gcc4.6/info --with-build-config=bootstrap-lto --enable-stage1-languages=c,lto --enable-languages=c,c++,fortran,lto,objc,obj-c++,java --with-gmp=/sw --with-libiconv-prefix=/sw --with-ppl=/sw --with-cloog=/sw --with-mpc=/sw --with-system-zlib --x-includes=/usr/X11R6/include --x-libraries=/usr/X11R6/lib --program-suffix=-fsf-4.6 --enable-checking=yes --enable-cloog-backend=isl
Comment 2 Jack Howarth 2011-03-12 03:57:23 UTC
This bug also manifests itself in the testsuite results from a normal bootstrap on x86_64-apple-darwin10. Using...

Using built-in specs.
COLLECT_GCC=gcc-4
COLLECT_LTO_WRAPPER=/sw/lib/gcc4.6/libexec/gcc/x86_64-apple-darwin10.7.0/4.6.0/lto-wrapper
Target: x86_64-apple-darwin10.7.0
Configured with: ../gcc-4.6-20110311/configure --prefix=/sw --prefix=/sw/lib/gcc4.6 --mandir=/sw/share/man --infodir=/sw/lib/gcc4.6/info --enable-languages=c,c++,fortran,lto,objc,obj-c++,java --with-gmp=/sw --with-libiconv-prefix=/sw --with-ppl=/sw --with-cloog=/sw --with-mpc=/sw --with-system-zlib --x-includes=/usr/X11R6/include --x-libraries=/usr/X11R6/lib --program-suffix=-fsf-4.6 --enable-checking=yes --enable-cloog-backend=isl
Thread model: posix
gcc version 4.6.0 20110312 (experimental) (GCC) 

under Xcode 4.0, the gcc testsuite failures...


Executing on host: /sw/src/fink.build/gcc46-4.6.0-1000/darwin_objdir/gcc/xgcc -B/sw/src/fink.build/gcc46-4.6.0-1000/darwin_objdir/gcc/ /sw/src/fink.build/gcc46-4.6.0-1000/gcc-4.6-20110311/gcc/testsuite/gcc.dg/torture/builtin-attr-1.c   -O2 -flto -flto-partition=none  -ffast-math  -lm   -m32 -o builtin-attr-1.exe    (timeout = 300)
/var/tmp//ccXfLGdb.s:12468:FATAL:too many sections (maximum 255)^M
^M
compiler exited with status 1
output is:
/var/tmp//ccXfLGdb.s:12468:FATAL:too many sections (maximum 255)^M
^M

FAIL: gcc.dg/torture/builtin-attr-1.c  -O2 -flto -flto-partition=none  (test for excess errors)
Excess errors:
/var/tmp//ccXfLGdb.s:12468:FATAL:too many sections (maximum 255)

Executing on host: /sw/src/fink.build/gcc46-4.6.0-1000/darwin_objdir/gcc/xgcc -B/sw/src/fink.build/gcc46-4.6.0-1000/darwin_objdir/gcc/ /sw/src/fink.build/gcc46-4.6.0-1000/gcc-4.6-20110311/gcc/testsuite/gcc.dg/torture/builtin-attr-1.c   -O2 -flto  -ffast-math  -lm   -m32 -o builtin-attr-1.exe    (timeout = 300)
/var/tmp//ccrXUYco.s:12468:FATAL:too many sections (maximum 255)^M
^M
compiler exited with status 1
output is:
/var/tmp//ccrXUYco.s:12468:FATAL:too many sections (maximum 255)^M
^M

FAIL: gcc.dg/torture/builtin-attr-1.c  -O2 -flto  (test for excess errors)
Excess errors:
/var/tmp//ccrXUYco.s:12468:FATAL:too many sections (maximum 255)

appear. These failures also exist but are undetected under Xcode prior to 4.0.
Comment 3 Dominique d'Humieres 2011-03-12 09:11:09 UTC
This is also true for Xcode 3.2.6 and causes the following failures

FAIL: gcc.dg/torture/builtin-attr-1.c  -O2 -flto -flto-partition=none  (test for excess errors)
FAIL: gcc.dg/torture/builtin-attr-1.c  -O2 -flto  (test for excess errors)
FAIL: g++.dg/torture/pr31863.C  -O2 -flto -flto-partition=none  (test for excess errors)
FAIL: g++.dg/torture/pr31863.C  -O2 -flto  (test for excess errors)

for both -m32 and -m64.
Comment 4 Iain Sandoe 2011-03-12 09:56:07 UTC
confirmed that:
(a) we create too many sections for the gcc.dg/torture/builtin-attr-1.c testcase (598)
(b) that 'as' does not detect this @XCode 3.2.5

looking at the output of otool -l, we have all the sections preserved in the initial .o files;

I guess this is because 'as' places all the output in one segment and encodes the seg/sect information manually.

The next question is 'do we ever present too many sections to the final (post LTO) link?' 
(and, if so, why?)

[it might be that we can live with excess sections in the intermediate objects (the extras are all in GNU_LTO) - so long as those are removed before we try to carry out the final link].
Comment 5 Iain Sandoe 2011-03-12 11:31:37 UTC
to clarify/as a reminder;

-  this limitation (n_sect  8 bits) is the reason that GNU_LTO sections are kept separate and emitted at the end of the object file on Darwin.
 - there are no symbols in these sections, so they will not be referred to in the symtab - therefore, we should never overflow the section reference.

we need to investigate if something is broken in those assumptions - or whether 'as' is just telling us something we already knew.
Comment 6 Jack Howarth 2011-03-12 22:49:40 UTC
Opened radar Bug ID# 9126174 for this issue with builtin-attr-1.s attached as the example of the potential assembler bug in Xcode 3.2.6 and 4.0.1.
Comment 7 Francois-Xavier Coudert 2011-03-13 08:16:01 UTC
Blocks bootstrap on a primary target; shouldn't it be P1/critical?
Comment 8 Steven Bosscher 2011-03-13 09:59:30 UTC
(In reply to comment #7)
> Blocks bootstrap on a primary target; shouldn't it be P1/critical?

I don't think so, for the following reasons.

First of all, there are no primary targets in *-apple-darwin*. The only Apple target in the release criteria is i686-apple-darwin which is a secondary target
(xf. http://gcc.gnu.org/gcc-4.6/criteria.html).

But IMVHO this is not a problem in GCC. LTO worked before, and now does not work anymore due to a change in software that GCC has no control over. This is a regression in Apple XCode. Even now, I can't find anywhere in the documentation of the Mach-O file format that there is a limitation to 255 sections per segment
(xf. http://developer.apple.com/library/mac/#documentation/DeveloperTools/Conceptual/MachORuntime/Reference/reference.html). There is a limitation of 255 for sections with relocations, but that is not applicable with GCC's LTO sections, because there are no relocations.

Call me paranoid, but I find it a suspicious coincidence that XCode 3.2.5 accepts GCC's LTO model, but Xcode 3.2.6 and newer do not, and this change happens days before GCC's first release with LTO for Apple... The XCode developers know that  GCC can use more than 255 sections, because we explicitly discussed this with XCode developers when I developed LTO for Mach-O.
Comment 9 Iain Sandoe 2011-03-13 10:07:50 UTC
my +1 generally to comment #8,

We are (re)-discussing with the Apple developer for ld (obviously that can't be in this forum).

IMO, 'as' is trying to check "if someone tries to write a section > 255 into the symbol n_sect field"  which we won't do. 

It is just a little to simplistic to do this by a simple count of sections.

The Apple developers are really helpful guys - and I'm sure we'll get an answer soon (I don't think we need to be paranoid, I suspect an oversight rather than a release plan... ;-) ).

containerizing the LTO stuff would probably be much easier now that we have simple-object in libiberty --- but it doesn't seem to be a last-minute kinda job - better left for stage 1 and back-ported for 4.6.1 (my £0.02).
Comment 10 Steven Bosscher 2011-03-13 19:48:41 UTC
The easiest way to fix this is maybe to just have more than one GNU_LTO segment. AFAIU the limit of 255 sections is a limit per segment. It is not difficult to have multiple GNU_LTO segments, I could even hack that in before GCC 4.6.

Ian, what do you think?

Confirming with Apple and having an answer on public record would be recommended.
Comment 11 Iain Sandoe 2011-03-13 20:19:31 UTC
(In reply to comment #10)
> The easiest way to fix this is maybe to just have more than one GNU_LTO
> segment. AFAIU the limit of 255 sections is a limit per segment. It is not
> difficult to have multiple GNU_LTO segments, I could even hack that in before
> GCC 4.6.

I'm sure that someone who as already updated would be willing to test a proposed solution ..  (Jack, Dominique...).  
... my systems are going to remain at XCode 3.2.5 at present ...

In the process of re-reading the Mach-o doc.  I haven't found any reason (other than the nlist struct) to see a limitation - e.g. nsects is uint32_t in the segment header - so no problem there.

===

(speculation) I guess it would depend on how 'as' detects the perceived problem.
 
Since an intermediate object puts all sections into single anonymous segment, if the error is reported on a simple section header count, increasing the number of segments won't make any difference (pessimistic assessment).  

The sources don't seem to have been released so far, so can't look there for an answer (yet).

> Confirming with Apple and having an answer on public record would be
> recommended.

Will put this into the melting pot.
Comment 12 Iain Sandoe 2011-03-13 20:32:56 UTC
(In reply to comment #11)
> (In reply to comment #10)
> > The easiest way to fix this is maybe to just have more than one GNU_LTO
> > segment. AFAIU the limit of 255 sections is a limit per segment. It is not
> > difficult to have multiple GNU_LTO segments, I could even hack that in before
> > GCC 4.6.

hmmm. maybe not .. re-reading:

n_sect 
An integer specifying the number of the section that this symbol can be found in, or NO_SECT if the 
symbol is not to be found in any section of this image.  The sections are contiguously numbered across 
segments, starting from 1, according to the order they appear in the LC_SEGMENT load commands. 

contiguously numbered across segments; seems to indicate that the section count is per file - not per segment.
Comment 13 Steven Bosscher 2011-03-13 20:41:12 UTC
Alright, multiple segments will not work. Or even if they do, it is another solution that may or may not work in the future depending on the whims of Apple.

So, a rewrite it will have to be, then.
Comment 14 Iain Sandoe 2011-03-13 20:54:03 UTC
(In reply to comment #13)
> Alright, multiple segments will not work. Or even if they do, it is another
> solution that may or may not work in the future depending on the whims of
> Apple.
> 
> So, a rewrite it will have to be, then.

the response from Nick is a confirmation of both points:

"The suggestion of using different segment names will not work.  Segments aren't really used in mach-o object files.   All mach-o object files have exactly one LC_SEGMENT load command.  It is final linked images that support multiple  LC_SEGMENT load commands.

It is possible to fix this in Xcode 4.x, but I think a solution that just uses one section for all gcc LTO data (and have your own sub-section scheme) will work with all darwin assembler versions."

==

I can only hope that the simple_object interface in libiberty will make this possible without introducing a dependency on libelf (which seems a little OTT for this relatively small requirement).
Comment 15 Jack Howarth 2011-03-13 21:54:17 UTC
FYI, it it comes down to having a dependency on libelf or heavily lobbying Apple to fix the broke change made to the assembler in Xcode 4.0.1 and 4.1, I would go with the second option. After all, they currently aren't properly honoring their own documented mach-o specifications and who knows what grief this change will cause for other software developers who have relied on this behavior.
Comment 16 Jack Howarth 2011-03-13 22:45:57 UTC
Actually the situation with Xcode 3.2.5 is pretty grim as well. Currently only Xcode 3.2.2 is available for download from connect.apple.com. So people needing to do new installs are trapped at the release since Software Update will only offer to take you to Xcode 3.2.6. This also that it will be trivial for a user currently on Xcode 3.2.5 to accidentally, via Software Update, take their machine to Xcode 3.2.6 and not be able to get back to 3.2.5. Perhaps we should just disable lto again for gcc 4.6.0.
Comment 17 mrs@gcc.gnu.org 2011-03-14 02:47:53 UTC
Author: mrs
Date: Mon Mar 14 02:47:49 2011
New Revision: 170929

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=170929
Log:
2011-03-13  Jack Howarth  <howarth@bromo.med.uc.edu>

	    PR lto/48086
	    * configure.ac: Disable LTO on darwin due to an assembler change in
	    Xcode 3.2.6/4.0 that limits the total number of sections/segments to
	    under 256.
	    * configure: Regenerate.

Modified:
    trunk/ChangeLog
    trunk/configure
    trunk/configure.ac
Comment 18 mrs@gcc.gnu.org 2011-03-14 02:54:37 UTC
Fixed.
Comment 19 Jack Howarth 2011-03-14 02:57:50 UTC
Should I create a new PR for following the progress of the re-implementation of LTO on darwin?
Comment 20 Mike Stump 2011-03-14 03:09:08 UTC
I'm ambivalent.  If people developing or following would like one, feel free to create one.  Depending on how safe it is, we could put it in sooner, and by that time, we'd need one.
Comment 21 Jack Howarth 2011-03-14 06:10:09 UTC
We also need to scratch the section of http://gcc.gnu.org/gcc-4.6/changes.html under Darwin/Mac OS X which says...

LTO-support.
Darwin has benefited from ongoing work on LTO; support for this is now stable and enabled by default.
Comment 22 mrs@gcc.gnu.org 2011-03-16 18:19:16 UTC
Author: mrs
Date: Wed Mar 16 18:19:12 2011
New Revision: 171058

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=171058
Log:
2011-03-16  Jack Howarth  <howarth@bromo.med.uc.edu>

	PR lto/48086
	* configure.ac: Re-enable LTO on *-apple-darwin9.
	* configure: Regenerate.

Modified:
    trunk/ChangeLog
    trunk/configure
    trunk/configure.ac
Comment 23 mrs@gcc.gnu.org 2011-03-16 18:27:41 UTC
Author: mrs
Date: Wed Mar 16 18:27:36 2011
New Revision: 171059

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=171059
Log:
2011-03-16  Jack Howarth  <howarth@bromo.med.uc.edu>

	PR lto/48086
	* configure.ac: Re-enable LTO on *-apple-darwin9.
	* configure: Regenerate.

Modified:
    branches/gcc-4_6-branch/ChangeLog
    branches/gcc-4_6-branch/configure
    branches/gcc-4_6-branch/configure.ac
Comment 24 mrs@gcc.gnu.org 2011-04-18 21:27:04 UTC
Author: mrs
Date: Mon Apr 18 21:27:00 2011
New Revision: 172671

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=172671
Log:
2011-04-18  Jack Howarth  <howarth@bromo.med.uc.edu>

	    PR lto/48086
	    * configure.ac: Re-enable LTO on *-apple-darwin9*.
	    * configure: Regenerate.

Modified:
    trunk/ChangeLog
    trunk/configure
    trunk/configure.ac
Comment 25 Dominique d'Humieres 2011-05-30 09:11:59 UTC
*** Bug 49228 has been marked as a duplicate of this bug. ***