Bug 48108 - lto should be containerized in a single mach-o section on darwin
lto should be containerized in a single mach-o section on darwin
Status: RESOLVED FIXED
Product: gcc
Classification: Unclassified
Component: target
4.6.0
: P3 normal
: ---
Assigned To: Not yet assigned to anyone
: lto
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-03-14 03:06 UTC by Jack Howarth
Modified: 2011-11-13 14:00 UTC (History)
7 users (show)

See Also:
Host: *-apple-darwin*
Target: *-apple-darwin*
Build: *-apple-darwin*
Known to work:
Known to fail:
Last reconfirmed: 2011-05-30 15:30:05


Attachments
Iain's work in progress for LTO containerization (14.33 KB, text/plain)
2011-05-30 15:16 UTC, Jack Howarth
Details
updated Work In Progress - ppc asm fix (24.09 KB, patch)
2011-07-07 10:23 UTC, Iain Sandoe
Details | Diff
updated work in progress (21.93 KB, patch)
2011-07-08 10:33 UTC, Iain Sandoe
Details | Diff
update WIP 2 (21.92 KB, patch)
2011-08-09 12:56 UTC, Iain Sandoe
Details | Diff
wip 3 (35.41 KB, patch)
2011-09-25 09:49 UTC, Iain Sandoe
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jack Howarth 2011-03-14 03:06:42 UTC
The LTO support on darwin has been disabled...

Author: mrs
Date: Mon Mar 14 02:47:49 2011
New Revision: 170929

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=170929
Log:
2011-03-13  Jack Howarth  <howarth@bromo.med.uc.edu>

        PR lto/48086
        * configure.ac: Disable LTO on darwin due to an assembler change in
        Xcode 3.2.6/4.0 that limits the total number of sections/segments to
        under 256.
        * configure: Regenerate.

Modified:
    trunk/ChangeLog
    trunk/configure
    trunk/configure.ac

due to the mishandling of <rdar://problem/7920267>, possible assembler bug exposed by LTO, by the Apple assembler developer where a solid 255 limit has been now imposed on all the sections= possible in a mach-o object file. This breaks the current implementation in gcc/lto/lto-object.c which relied on the ability to have an unlimited number of symbol-less GNU_LTO sections at the end of the object file.
   The darwin LTO support in  gcc/lto/lto-object.c  and gcc/config/darwin.c needs to be written to containerize all of these GNU_LTO sections into a single mach-o section. Hopefully this can be done without having to resort to an elf container, which would introduce an undesired dependency on libelf for FSF gcc on darwin.
Comment 1 Jack Howarth 2011-03-14 03:10:24 UTC
Note that LTO was disabled for gcc 4.6.0 due to the inability of end-users to obtain an appropriate Xcode 3.2.5 now that the broken Xcode 3.2.6/4.0 have been released.
Comment 2 Mike Stump 2011-03-14 03:13:27 UTC
Another fix might be to have pure elf .o files...  ld I think will read elf .o files...  [shhhh]   Don't tell anyone I said that.  If not, we might be able to get Apple to do that.  This might then require an FSF binutils.
Comment 3 Iain Sandoe 2011-03-14 09:00:52 UTC
(In reply to comment #2)
> Another fix might be to have pure elf .o files...  ld I think will read elf .o
> files...  [shhhh]   Don't tell anyone I said that.  If not, we might be able to
> get Apple to do that.  This might then require an FSF binutils.

In the short term, a wrapper around the LTO stuff with a more sophisticated index seems a lot easier.

However, it is in the back of my mind to split the machopic stuff out of darwin* somewhen in 4.7 so that darwin sections could be used in an elf (or other supporting named sections) container (and give us a route for removing the conditionalization on TARGET_MACHO from config/{i386,rs6000}/ )  

WDYT?
Comment 4 Mike Stump 2011-03-14 23:34:38 UTC
> WDYT?

Sounds fine to me.
Comment 5 Jack Howarth 2011-03-17 12:02:02 UTC
Testresults for gcc-4_6-branch built under Xcode 4.0 using lto-bootstrap/profiledbootstrap with Iain's first attempt at containerization.

http://gcc.gnu.org/ml/gcc-testresults/2011-03/msg01695.html
Comment 6 vincenzo Innocente 2011-05-30 12:01:11 UTC
this is actually a follow up of
http://gcc.gnu.org/bugzilla/show_bug.cgi?id=49228
that has been identified as a duplicate of this bug.

I applied the patch
now stops here
/bin/sh ./libtool --tag=CC   --mode=compile /Users/innocent/RealStuff/gcc-4.7-20110528/host-x86_64-apple-darwin10.7.0/gcc/xgcc -B/Users/innocent/RealStuff/gcc-4.7-20110528/host-x86_64-apple-darwin10.7.0/gcc/ -B/usr/local/x86_64-apple-darwin10.7.0/bin/ -B/usr/local/x86_64-apple-darwin10.7.0/lib/ -isystem /usr/local/x86_64-apple-darwin10.7.0/include -isystem /usr/local/x86_64-apple-darwin10.7.0/sys-include    -DHAVE_CONFIG_H -I. -I../.././libgfortran  -iquote../.././libgfortran/io -I../.././libgfortran/../gcc -I../.././libgfortran/../gcc/config -I../.././libgfortran/../libquadmath -I../../host-x86_64-apple-darwin10.7.0/gcc  -std=gnu99 -Wall -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition -Wextra -Wwrite-strings -fcx-fortran-rules -ffunction-sections -fdata-sections  -g -O2 -ftree-vectorize -fPIC -MT fmain.lo -MD -MP -MF .deps/fmain.Tpo -c -o fmain.lo ../.././libgfortran/fmain.c
libtool: compile:  /Users/innocent/RealStuff/gcc-4.7-20110528/host-x86_64-apple-darwin10.7.0/gcc/xgcc -B/Users/innocent/RealStuff/gcc-4.7-20110528/host-x86_64-apple-darwin10.7.0/gcc/ -B/usr/local/x86_64-apple-darwin10.7.0/bin/ -B/usr/local/x86_64-apple-darwin10.7.0/lib/ -isystem /usr/local/x86_64-apple-darwin10.7.0/include -isystem /usr/local/x86_64-apple-darwin10.7.0/sys-include -DHAVE_CONFIG_H -I. -I../.././libgfortran -iquote../.././libgfortran/io -I../.././libgfortran/../gcc -I../.././libgfortran/../gcc/config -I../.././libgfortran/../libquadmath -I../../host-x86_64-apple-darwin10.7.0/gcc -std=gnu99 -Wall -Wstrict-prototypes -Wmissing-prototypes -Wold-style-definition -Wextra -Wwrite-strings -fcx-fortran-rules -ffunction-sections -fdata-sections -g -O2 -ftree-vectorize -fPIC -MT fmain.lo -MD -MP -MF .deps/fmain.Tpo -c ../.././libgfortran/fmain.c  -fno-common -DPIC -o .libs/fmain.o
In file included from ../.././libgfortran/libgfortran.h:53:0,
                 from ../.././libgfortran/fmain.c:4:
../.././libgfortran/../libquadmath/quadmath_weak.h:39:1: internal compiler error: tree check: expected tree that contains ‘common’ structure, have ‘identifier_node’ in assemble_alias, at varasm.c:5789
Please submit a full bug report,
with preprocessed source if appropriate.
See <http://gcc.gnu.org/bugs.html> for instructions.
Comment 7 Dominique d'Humieres 2011-05-30 12:19:45 UTC
This is pr49190. If you want to proceed, you need to revert revision 174286 (or to wait for a fix).
BTW which version of Xcode are you using?
Comment 8 vincenzo Innocente 2011-05-30 12:26:37 UTC
On 30 May, 2011, at 2:20 PM, dominiq at lps dot ens.fr wrote:

> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48108
> 
> --- Comment #7 from Dominique d'Humieres <dominiq at lps dot ens.fr> 2011-05-30 12:19:45 UTC ---
> This is pr49190. If you want to proceed, you need to revert revision 174286 (or
> to wait for a fix).
ok, I will wait for a fix
it built g++ and that's enough for let me going for a while :-)
> BTW which version of Xcode are you using?
apparently Version 3.2.6

thanks for the quick reply

v.
Comment 9 Dominique d'Humieres 2011-05-30 13:04:29 UTC
> ok, I will wait for a fix

Meanwhile you can also configure with --enable-checking=release.
Comment 10 Jack Howarth 2011-05-30 15:15:31 UTC
I have been using Iain's last work in progress patch and it works fine with both gcc 4.6 branch and gcc trunk. Hopefully he can get back to polishing it or someone else can pick up the patch and complete it.
Comment 11 Jack Howarth 2011-05-30 15:16:25 UTC
Created attachment 24397 [details]
Iain's work in progress for LTO containerization
Comment 12 Iain Sandoe 2011-05-30 15:30:05 UTC
(In reply to comment #11)
> Created attachment 24397 [details]
> Iain's work in progress for LTO containerization

Sorry that I can't commit any time to GCC right now.

The main outstanding issue with this patch is that the intermediate files created by GCC are still unbounded in the number of sections. 

So long as the only consumer of those files is GCC, no problem (since the arrangement has been made to ensure that relocatable sections come first).

However, those intermediate files are still technically 'wrong' and therefore the writer should be updated to do the same encapsulation.  Once that is done there will be no need to retain the ability to recognize GCC vs 'normal' objects.
Comment 13 Iain Sandoe 2011-07-07 10:23:06 UTC
Created attachment 24705 [details]
updated Work In Progress - ppc asm fix

NOTE: this is still very much a "work in progress" and is neither complete, nor really ready for extensive testing...

... However, this minor mod fixes the issue noted with powerpc (comment introducers at the end of asm statements).

Patch against trunk - but should be similar for 4.6.x.
Comment 14 Iain Sandoe 2011-07-08 10:33:47 UTC
Created attachment 24712 [details]
updated work in progress

there were a couple of hunks in the previous from another LTO patch, that reportedly gave problems on x86-64-darwin11.   The attached is OK AFAICT, on 4.6, trunk x86-64-darwin10, *-darwin9.

(still incomplete).
Comment 15 Jack Howarth 2011-07-08 11:18:25 UTC
(In reply to comment #14)
> Created attachment 24712 [details]
> updated work in progress
> 
> there were a couple of hunks in the previous from another LTO patch, that
> reportedly gave problems on x86-64-darwin11.   The attached is OK AFAICT, on
> 4.6, trunk x86-64-darwin10, *-darwin9.
> 
> (still incomplete).

http://gcc.gnu.org/ml/gcc-testresults/2011-07/msg00877.html

No serious regressions other than objc and friends when applied to gcc 4.6.1.
Comment 16 David Fang 2011-07-10 04:41:14 UTC
This current patch yields excellent results on powerpc-darwin8 now:
http://gcc.gnu.org/ml/gcc-testresults/2011-07/msg01092.html
Comment 17 Jack Howarth 2011-08-05 18:52:05 UTC
(In reply to comment #12)
> (In reply to comment #11)
> > Created attachment 24397 [details]
> > Iain's work in progress for LTO containerization
> 
> Sorry that I can't commit any time to GCC right now.
> 
> The main outstanding issue with this patch is that the intermediate files
> created by GCC are still unbounded in the number of sections. 
> 
> So long as the only consumer of those files is GCC, no problem (since the
> arrangement has been made to ensure that relocatable sections come first).
> 
> However, those intermediate files are still technically 'wrong' and therefore
> the writer should be updated to do the same encapsulation.  Once that is done
> there will be no need to retain the ability to recognize GCC vs 'normal'
> objects.

Should we really be so hung up on the intermediate file issue? After all, why would anyone try to use object files generated with -flto with a compiler that doesn't support -flto. I would argue that this prevents users from accidentally disabling the -flto in that case (where they erroneously use a non-LTO aware compiler). This could simply be marked as a TODO.
Comment 18 Iain Sandoe 2011-08-09 12:56:50 UTC
Created attachment 24958 [details]
update WIP 2

testing lto bootstraps together with compare debug, revealed that the $gnu$lto... symbols used to determine the sizes etc. should have been made local (i.e. L$gnu$lto..).

still need to fix simple-object output to DTRT.
Comment 19 Iain Sandoe 2011-09-25 09:49:04 UTC
Created attachment 25359 [details]
wip 3

The new attachment implements the 'wrapper' in simple-object read and write and darwin.c

Modulo checking comments/white space it's a candidate for fixing the bug.

I am not sure why this has been labelled 'enhancement' - we are generating badly-formed mach-o (according to the vendor) - the fact that we don't see a reported error with earlier vendor tool-chains doesn't seem to make it an 'enhancement' to get it right ;)

---

I've done --with-build-config='bootstrap-lto bootstrap-debug' (no Ada on trunk because of unrelated issues) on:

trunk *-darwin9 (less Ada)
4.6 i686-darwin9 (incl. Ada) 4.6 powerpc-darwin9 is running.
trunk x86_64-darwin10 (less Ada)
4.6 x86_64-darwin10 (incl. Ada)

So - darwin 11 needs checking.
The patch applies cleanly to both 4.6 and trunk as of this time.
Comment 20 Jack Howarth 2011-09-26 00:47:32 UTC
(In reply to comment #19)
> Created attachment 25359 [details]
> wip 3
> 

Regtest on darwin11 at http://gcc.gnu.org/ml/gcc-testresults/2011-09/msg02611.html.
Comment 21 Iain Sandoe 2011-10-26 12:38:05 UTC
Author: iains
Date: Wed Oct 26 12:37:59 2011
New Revision: 180523

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=180523
Log:

gcc:

	PR target/48108
	* config/darwin.c (top level): Amend comments concerning LTO output.
	(lto_section_num): New variable.  (darwin_lto_section_e): New GTY.
	(LTO_SECTS_SECTION, LTO_INDEX_SECTION): New.
	(LTO_NAMES_SECTION): Rename.
	(darwin_asm_named_section): Record LTO section counts and switches
	in a vec of darwin_lto_section_e.
	(darwin_file_start): Remove unused code.
	(darwin_file_end): Put an LTO section termination label.  Handle
	output of the wrapped LTO sections, index and names table.

libiberty:

	PR target/48108
	* simple-object-mach-o.c  (GNU_WRAPPER_SECTS, GNU_WRAPPER_INDEX,
	GNU_WRAPPER_NAMES): New macros.
	(simple_object_mach_o_segment): Handle wrapper scheme.
	(simple_object_mach_o_write_section_header): Allow the segment name
	to be supplied.
	(simple_object_mach_o_write_segment): Handle wrapper scheme.  Ensure
	that the top-level segment name in the load command is empty.
	(simple_object_mach_o_write_to_file): Determine the number of
	sections during segment output, use that in writing the header.


Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/config/darwin.c
    trunk/libiberty/ChangeLog
    trunk/libiberty/simple-object-mach-o.c
Comment 22 Jack Howarth 2011-10-26 13:28:20 UTC
Can we backport r180523 to gcc-4_6-branch (perhaps even for gcc 4.6.2)?
Also in gcc trunk we should also now fully revert...

Author: mrs
Date: Mon Mar 14 02:47:49 2011
New Revision: 170929

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=170929
Log:
2011-03-13  Jack Howarth  <howarth@bromo.med.uc.edu>

	    PR lto/48086
	    * configure.ac: Disable LTO on darwin due to an assembler change in
	    Xcode 3.2.6/4.0 that limits the total number of sections/segments to
	    under 256.
	    * configure: Regenerate.

Modified:
    trunk/ChangeLog
    trunk/configure
    trunk/configure.ac

since PR48086 is solved by r180523.
Comment 23 Iain Sandoe 2011-11-13 13:58:47 UTC
Author: iains
Date: Sun Nov 13 13:58:43 2011
New Revision: 181336

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=181336
Log:

gcc:

	PR target/48108
	Backport from mainline r180523
	* config/darwin.c (top level): Amend comments concerning LTO output.
	(lto_section_num): New variable.  (darwin_lto_section_e): New GTY.
	(LTO_SECTS_SECTION, LTO_INDEX_SECTION): New.
	(LTO_NAMES_SECTION): Rename.
	(darwin_asm_named_section): Record LTO section counts and switches
	in a vec of darwin_lto_section_e.
	(darwin_file_start): Remove unused code.
	(darwin_file_end): Put an LTO section termination label.  Handle
	output of the wrapped LTO sections, index and names table.

libiberty:

	PR target/48108
	Backport from mainline r180523
	* simple-object-mach-o.c  (GNU_WRAPPER_SECTS, GNU_WRAPPER_INDEX,
	GNU_WRAPPER_NAMES): New macros.
	(simple_object_mach_o_segment): Handle wrapper scheme.
	(simple_object_mach_o_write_section_header): Allow the segment name
	to be supplied.
	(simple_object_mach_o_write_segment): Handle wrapper scheme.  Ensure
	that the top-level segment name in the load command is empty.
	(simple_object_mach_o_write_to_file): Determine the number of
	sections during segment output, use that in writing the header.


Modified:
    branches/gcc-4_6-branch/gcc/ChangeLog
    branches/gcc-4_6-branch/gcc/config/darwin.c
    branches/gcc-4_6-branch/libiberty/ChangeLog
    branches/gcc-4_6-branch/libiberty/simple-object-mach-o.c
Comment 24 Iain Sandoe 2011-11-13 14:00:07 UTC
(In reply to comment #22)
> Can we backport r180523 to gcc-4_6-branch (perhaps even for gcc 4.6.2)?

done now - Mike had approved on the mailing list.

> Also in gcc trunk we should also now fully revert...
> 
> Author: mrs
> Date: Mon Mar 14 02:47:49 2011
> New Revision: 170929

I don't think we can do this until PR49992 and likely PR48109 are fixed.

Fixed.