Created attachment 27345 [details] directory containing three (very similar) real-life file after pre-compiler step in the following example (not much reduced from a real-life code) a symbol, very similar to others, is made hidden for no good reason. Apparently it also depends on the order of the input files… bzip2 -d ltobug3.tar.bz2 tar -xf ltobug3.tar cd ltobug3/ c++ -fvisibility-inlines-hidden -O2 -flto -shared -fPIC -o bha.so d1.cc d2.cc d3.cc ; nm -C bha.so | grep "makeKey" | grep -v "type" 0000000000004290 t _ZN3edm10eventsetup15heterocontainer7makeKeyI11DummyRecordNS0_19EventSetupRecordKeyEEET0_v.local.77.4195 0000000000004750 W edm::eventsetup::EventSetupRecordKey edm::eventsetup::heterocontainer::makeKey<Dummy2Record, edm::eventsetup::EventSetupRecordKey>() 00000000000048a0 W edm::eventsetup::EventSetupRecordKey edm::eventsetup::heterocontainer::makeKey<DepRecord, edm::eventsetup::EventSetupRecordKey>() c++ -fvisibility-inlines-hidden -O2 -flto -shared -fPIC -o bha.so d2.cc d1.cc d3.cc ; nm -C bha.so | grep "makeKey" | grep -v "type" 00000000000048a0 W edm::eventsetup::EventSetupRecordKey edm::eventsetup::heterocontainer::makeKey<DummyRecord, edm::eventsetup::EventSetupRecordKey>() 0000000000004750 W edm::eventsetup::EventSetupRecordKey edm::eventsetup::heterocontainer::makeKey<Dummy2Record, edm::eventsetup::EventSetupRecordKey>() 00000000000049f0 W edm::eventsetup::EventSetupRecordKey edm::eventsetup::heterocontainer::makeKey<DepRecord, edm::eventsetup::EventSetupRecordKey>()
This has nothing to do with -fvisbility-inlines-hidden (well, that might be necessary to trigger it). LTO brings symbols local to ship them to multiple partitions and partition layout (unfortunately, but probably also unavoidably) depends on linker file order.
understood why it is related to the order of the input files. Still that particular symbol shall not be hidden. I've a library with 307 of those symbols, and lto hides only 7 of them!
LTO works in a way that it first collect whole program and promote all symbols that are not externally visible static. After doing whole program optimization the program is decomposed again into ltrans unit and compiled independently. Symbols used from multiple ltrans unit are made hidden, so the units can refer to it. In your program apparently only those few symbols are crossing boundary of ltrans. So this is really independent thing. You can try to compile with -flto-partition=none. It will take longer but should avoid the hidden symbols. File order independency of partitioning is possible, but I doubt it is desirable design goal. In fact keeping things approximately in the order it was passed to linker makes sense unless compiler knows better. Honza
Hi Honza, I forgot to say that I tried both -flto-partition=1to1 and -flto-partition=none with the same result the point is that the symbols in question refers to a templated function explicitly defined as template edm::eventsetup::EventSetupRecordKey edm::eventsetup::heterocontainer::makeKey<_recordclassname_, edm::eventsetup::EventSetupRecordKey>() in the attached there are three of those: only one is hidden. In the "real" library there are 307 of those, only 7 are hidden! SO I suspect some sort of bug in lto I will try to produce a synthetic test
(In reply to comment #4) > Hi Honza, > I forgot to say that I tried both > -flto-partition=1to1 > and > -flto-partition=none > with the same result > > the point is that the symbols in question refers to a templated function > explicitly defined as > template edm::eventsetup::EventSetupRecordKey > edm::eventsetup::heterocontainer::makeKey<_recordclassname_, > edm::eventsetup::EventSetupRecordKey>() > > in the attached there are three of those: only one is hidden. > In the "real" library there are 307 of those, only 7 are hidden! > SO I suspect some sort of bug in lto > > I will try to produce a synthetic test All hidden symbols are postfixed with something like .local.77.4195, making them no longer the symbols for the explicitely instantiated functions. I suppose you say that the bug is that you have explicitely instantiated some templates which you want to have appear in the library as exported but LTO makes all instances hidden so that no externally visible symbol for that explicitely instantiated template remains?
On 9 May, 2012, at 11:10 AM, rguenth at gcc dot gnu.org wrote: > All hidden symbols are postfixed with something like .local.77.4195, making > them no longer the symbols for the explicitely instantiated functions. > > I suppose you say that the bug is that you have explicitely instantiated > some templates which you want to have appear in the library as exported > but LTO makes all instances hidden so that no externally visible symbol > for that explicitely instantiated template remains? correct. The additional point is that this happens only for some symbols, not all, even if the definitions/declarations are identical.
(In reply to comment #6) > On 9 May, 2012, at 11:10 AM, rguenth at gcc dot gnu.org wrote: > > > All hidden symbols are postfixed with something like .local.77.4195, making > > them no longer the symbols for the explicitely instantiated functions. > > > > I suppose you say that the bug is that you have explicitely instantiated > > some templates which you want to have appear in the library as exported > > but LTO makes all instances hidden so that no externally visible symbol > > for that explicitely instantiated template remains? > > correct. > The additional point is that this happens only for some symbols, not all, even > if the definitions/declarations are identical. Ok, so the question would be - why does GCC think this symbol is not possibly referenced from outside of the LTO unit?
On 9 May, 2012, at 11:58 AM, rguenth at gcc dot gnu.org wrote: > Ok, so the question would be - why does GCC think this symbol is not > possibly referenced from outside of the LTO unit? Indeed.
Btw, the testcase does not reproduce for me. What binutils version and which linker do you use? Do you actually end up using -fuse-linker-plugin? Because it reproduces for me with -fno-use-linker-plugin only (a path that is probably not very well tested these days ...)
On 9 May, 2012, at 12:15 PM, rguenth at gcc dot gnu.org wrote: > http://gcc.gnu.org/bugzilla/show_bug.cgi?id=53282 > > --- Comment #9 from Richard Guenther <rguenth at gcc dot gnu.org> 2012-05-09 10:15:21 UTC --- > Btw, the testcase does not reproduce for me. What binutils version and which > linker do you use? Do you actually end up using -fuse-linker-plugin? Because > it reproduces for me with -fno-use-linker-plugin only (a path that is probably > not very well tested these days …)we use ld -v GNU gold (GNU Binutils 2.21.1) 1.11 and we do not use explicitely -fuse-linker-plugin, gcc is configured as follow c++ -v Using built-in specs. COLLECT_GCC=c++ COLLECT_LTO_WRAPPER=/afs/cern.ch/user/i/innocent/w3/gcc47slc5/libexec/gcc/x86_64-unknown-linux-gnu/4.7.1/lto-wrapper Target: x86_64-unknown-linux-gnu Configured with: ./configure --enable-languages=c,c++,fortran --disable-multilib --enable-gold=yes --disable-nls --enable-lto --with-mpc=/afs/cern.ch/cms/slc5_amd64_gcc470/external/gcc/4.7.0 --with-gmp=/afs/cern.ch/cms/slc5_amd64_gcc470/external/gcc/4.7.0 --with-mpfr=/afs/cern.ch/cms/slc5_amd64_gcc470/external/gcc/4.7.0 --prefix=/afs/cern.ch/user/i/innocent/w3/gcc47slc5 : (reconfigured) ./configure --enable-languages=c,c++,fortran --disable-multilib --enable-gold=yes --disable-nls --enable-lto --with-mpc=/afs/cern.ch/cms/slc5_amd64_gcc470/external/gcc/4.7.0 --with-gmp=/afs/cern.ch/cms/slc5_amd64_gcc470/external/gcc/4.7.0 --with-mpfr=/afs/cern.ch/cms/slc5_amd64_gcc470/external/gcc/4.7.0 --prefix=/afs/cern.ch/user/i/innocent/w3/gcc47slc5 Thread model: posix gcc version 4.7.1 20120508 (prerelease) [gcc-4_7-branch revision 187276] (GCC) >
if I add -fuse-linker-plugin I get c++: error: -fuse-linker-plugin is not supported in this configuration my understanding was that since 4.6 linker-plugin was not required anymore for lto
> if I add -fuse-linker-plugin I get > c++: error: -fuse-linker-plugin is not supported in this configuration > > my understanding was that since 4.6 linker-plugin was not required anymore for > lto No, LTO plugin is needed for "proper" LTO and will always be. We have alternative non-plugin path that results in poorer code quality. In non-plugin path there is logic to privatize comdats whose address is not taken, since they can be duplicated. With explicit instantiation this code however should not trigger. I will check. Honza
On 9 May, 2012, at 2:21 PM, hubicka at ucw dot cz wrote: > No, LTO plugin is needed for "proper" LTO and will always be. We have > alternative > non-plugin path that results in poorer code quality. I see. could you please specify the configure options to be used ? I tried --enable-linker-plugin and does not help thois is how I configured latest version of the trunk c++ -v Using built-in specs. COLLECT_GCC=c++ COLLECT_LTO_WRAPPER=/afs/cern.ch/user/i/innocent/w2/libexec/gcc/x86_64-unknown-linux-gnu/4.8.0/lto-wrapper Target: x86_64-unknown-linux-gnu Configured with: ./configure --prefix=/afs/cern.ch/user/i/innocent/w2 --enable-languages=c,c++,fortran -enable-gold=yes --enable-lto --with-build-config=bootstrap-lto --with-gmp-lib=/usr/local/lib64 --with-mpfr-lib=/usr/local/lib64 -with-mpc-lib=/usr/local/lib64 --enable-cloog-backend=isl --with-cloog=/usr/local --with-ppl-lib=/usr/local/lib64 CFLAGS='-O2 -ftree-vectorize -fPIC' CXXFLAGS='-O2 -fPIC -ftree-vectorize -fvisibility-inlines-hidden -march=native' -enable-libitm -disable-multilib --enable-linker-plugin Thread model: posix gcc version 4.8.0 20120509 (experimental) [trunk revision 187326] (GCC) and I get c++ -fuse-linker-plugin -fvisibility-inlines-hidden -O2 -flto -shared -fPIC -o bha.so d1.cc d2.cc d3.cc c++: error: -fuse-linker-plugin is not supported in this configuration
(In reply to comment #11) > if I add -fuse-linker-plugin I get > c++: error: -fuse-linker-plugin is not supported in this configuration > > my understanding was that since 4.6 linker-plugin was not required anymore for > lto Since GCC 4.6 a linker-plugin is used automatically if a suitable linker is found. If that doesn't work for you that means the configure check "linker plugin support" fails - you have to check why. Did you build binutils with support for plugins?
I'm indeed using a "ld" that is not the one of the system. googling around I found this: http://comments.gmane.org/gmane.comp.gcc.help/41304 titled" g++: error: -fuse-linker-plugin is not supported in this configuration" at some point it hints to the fact the gcc configure seems to ignore the ld in the path and uses the system one, detects no plugin etc. I configured gcc with --with-ld=/usr/local/bin/ld and it does not help LD=/usr/local/bin/ld neither. I'm sure that plugin is supported by this version of ld. I wil try more, any suggestion is appreciated. in the mean time we can change the bug report saying that it happens if the linker-plugin is not used…
(In reply to comment #15) > I'm indeed using a "ld" that is not the one of the system. > > googling around I found this: > http://comments.gmane.org/gmane.comp.gcc.help/41304 > titled" g++: error: -fuse-linker-plugin is not supported in this configuration" > at some point it hints to the fact the gcc configure seems to ignore the ld in > the path > and uses the system one, detects no plugin etc. > > > I configured gcc with --with-ld=/usr/local/bin/ld and it does not help > LD=/usr/local/bin/ld neither. > > I'm sure that plugin is supported by this version of ld. > I wil try more, any suggestion is appreciated. > > > in the mean time we can change the bug report saying that it happens if the > linker-plugin is not used… I think you need to use --with-plugin-ld. Looking at how the "default" linker is determined it seems you would need to set DEFAULT_LINKER, at least I cannot see how --with-ld would end up in $gcc_cv_ld (configure would need to print your supplied ld when looking for ld and for the ld to use for the target).
I managed to get the linker-plugin enabled 1) building gcc in a directory different than the svn source one 2) using --with-build-time-tools= to point where we have all "current" tool-set now I see grep "linker plugin " build.log checking linker plugin support... 2 checking linker plugin support... 2 checking linker plugin support... 2 when previously was … 0 now we are fighting to install the new binutils above an old distro… thanks for all the support given so far