Copy relocation in PIE is incompatible with DSO created with -Wl,-Bsymbolic: [hjl@gnu-tools-1 copyreloc-2]$ cat x.c extern int a; extern int *a_p (void); extern void bar (void); extern void *bar_p (void); int main() { if (bar_p () != &bar) __builtin_abort(); if (a_p () != &a) __builtin_abort(); bar (); if (a != 30) __builtin_abort(); return 0; } [hjl@gnu-tools-1 copyreloc-2]$ cat bar.c int a; void bar () { a = 30; } void * bar_p () { return &bar; } int * a_p () { return &a; } [hjl@gnu-tools-1 copyreloc-2]$ make /export/build/gnu/gcc-5/build-x86_64-linux/gcc/xgcc -B/export/build/gnu/gcc-5/build-x86_64-linux/gcc/ -g -O2 -fPIE -c -o x.o x.c /export/build/gnu/gcc-5/build-x86_64-linux/gcc/xgcc -B/export/build/gnu/gcc-5/build-x86_64-linux/gcc/ -g -O2 -fPIC -c -o bar.o bar.c /export/build/gnu/gcc-5/build-x86_64-linux/gcc/xgcc -B/export/build/gnu/gcc-5/build-x86_64-linux/gcc/ -Wl,-Bsymbolic -shared -o libbar.so bar.o /export/build/gnu/gcc-5/build-x86_64-linux/gcc/xgcc -B/export/build/gnu/gcc-5/build-x86_64-linux/gcc/ -pie -o x x.o libbar.so -Wl,-rpath,. ./x Makefile:13: recipe for target 'all' failed make: *** [all] Aborted (core dumped) [hjl@gnu-tools-1 copyreloc-2]$
A patch is posted at https://gcc.gnu.org/ml/gcc-patches/2015-04/msg01546.html
*** Bug 65888 has been marked as a duplicate of this bug. ***
Thanks H.J.! Can I ask that -fsymbolic be the default? Otherwise, code with -fPIE MUST add -fsymbolic in GCC 5+, but can't add it prior because the option didn't exist. Please leave that for a release or two so that we can adapt buildsystems.
(In reply to Thiago Macieira from comment #3) > Thanks H.J.! > > Can I ask that -fsymbolic be the default? Otherwise, code with -fPIE MUST > add -fsymbolic in GCC 5+, but can't add it prior because the option didn't > exist. Please leave that for a release or two so that we can adapt Linux kernel has things like M16_CFLAGS := $(call cc-option, -m16, $(CODE16GCC_CFLAGS)) It adds -m16 only if it exits. Why can't KDE do something like this? There are many ways to achieve it and it works with all compilers. > buildsystems. By default, GCC should be psABI compliant. -Bsymbolic isn't psABI compliant since the resulting shared library doesn't work with normal executable. I don't believe -fsymbolic should be the default, just like that -Bsymbolic shouldn't be the default for linker either.
(In reply to Thiago Macieira from comment #3) > Thanks H.J.! > > Can I ask that -fsymbolic be the default? Otherwise, code with -fPIE MUST > add -fsymbolic in GCC 5+, but can't add it prior because the option didn't BTW, you should add -fsymbolic to -fPIC if the shared library will be created by -Bsymbolic. It will improve shared library performance.
(In reply to H.J. Lu from comment #5) > > BTW, you should add -fsymbolic to -fPIC if the shared library will be > created by -Bsymbolic. It will improve shared library performance. Here is an example: [hjl@gnu-tools-1 pr65886-lto]$ cat a.c int a = -1; [hjl@gnu-tools-1 pr65886-lto]$ cat b.c extern int a; void bar () { a = 30; } [hjl@gnu-tools-1 pr65886-lto]$ vi b.c [hjl@gnu-tools-1 pr65886-lto]$ cat a.c int a = -1; [hjl@gnu-tools-1 pr65886-lto]$ cat b.c extern int a; void bar () { a = 30; } [hjl@gnu-tools-1 pr65886-lto]$ make libb.so /export/build/gnu/gcc-5/build-x86_64-linux/gcc/xgcc -B/export/build/gnu/gcc-5/build-x86_64-linux/gcc/ -g -O2 -fsymbolic -fPIC -flto -c -o b.o b.c /export/build/gnu/gcc-5/build-x86_64-linux/gcc/xgcc -B/export/build/gnu/gcc-5/build-x86_64-linux/gcc/ -g -O2 -fsymbolic -fPIC -flto -c -o a.o a.c /export/build/gnu/gcc-5/build-x86_64-linux/gcc/xgcc -B/export/build/gnu/gcc-5/build-x86_64-linux/gcc/ -flto -Wl,-Bsymbolic -g -O2 -fsymbolic -shared -o libb.so b.o a.o [hjl@gnu-tools-1 pr65886-lto]$ gdb libb.so GNU gdb (GDB) Fedora 7.8.2-38.fc21 Copyright (C) 2014 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". Type "show configuration" for configuration details. For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. Find the GDB manual and other documentation resources online at: <http://www.gnu.org/software/gdb/documentation/>. For help, type "help". Type "apropos word" to search for commands related to "word"... Reading symbols from libb.so...done. (gdb) disass bar Dump of assembler code for function bar: 0x00000000000005e0 <+0>: movl $0x1e,0x2002be(%rip) # 0x2008a8 <a> 0x00000000000005ea <+10>: retq End of assembler dump. (gdb)
(In reply to H.J. Lu from comment #4) > (In reply to Thiago Macieira from comment #3) > > Thanks H.J.! > > > > Can I ask that -fsymbolic be the default? Otherwise, code with -fPIE MUST > > add -fsymbolic in GCC 5+, but can't add it prior because the option didn't > > exist. Please leave that for a release or two so that we can adapt > > Linux kernel has things like > > M16_CFLAGS := $(call cc-option, -m16, $(CODE16GCC_CFLAGS)) > > It adds -m16 only if it exits. Why can't KDE do something like this? > There are many ways to achieve it and it works with all compilers. We can. The problem is not the ability, it's the need to do it. No buildsystem currently has support for -fsymbolic, since it doesn't exist yet. However, the lack of such option will produce subtle bugs. I'm asking for a grace period until the changes propagate. I can add a runtime check to QtCore, though. > By default, GCC should be psABI compliant. -Bsymbolic isn't psABI compliant > since the resulting shared library doesn't work with normal executable. I > don't believe -fsymbolic should be the default, just like that -Bsymbolic > shouldn't be the default for linker either. I would argue that those should be the default and that we're optimising for the wrong thing. For example, in your email with the patch, you said: > Some experiments on google and SPEC CPU benchmarks show that the extra > instruction affects performance by 1% to 5%. You're not accounting for loss of performance in the shared libraries that can't use -Bsymbolic due to the copy relocations, both at load-time (relocations by name) and at runtime (indirect addressing to local symbols). Maybe our experiences with shared libraries isn't the same: mine is that libraries are 10x bigger and more complex than the applications using them. So I submit that your benchmarks are incomplete.
(In reply to H.J. Lu from comment #6) > (In reply to H.J. Lu from comment #5) > > > > BTW, you should add -fsymbolic to -fPIC if the shared library will be > > created by -Bsymbolic. It will improve shared library performance. Will do. > /export/build/gnu/gcc-5/build-x86_64-linux/gcc/xgcc > -B/export/build/gnu/gcc-5/build-x86_64-linux/gcc/ -flto -Wl,-Bsymbolic -g > -O2 -fsymbolic -shared -o libb.so b.o a.o Will the LTO be required here? Or will it work without LTO too? > Dump of assembler code for function bar: > 0x00000000000005e0 <+0>: movl $0x1e,0x2002be(%rip) # 0x2008a8 <a> > 0x00000000000005ea <+10>: retq > End of assembler dump. I've been accomplishing the same with protected visibility, but I keep running into linker bugs, so I haven't been able to enable it by default. Looks like the option here will be the proper solution.
(In reply to Thiago Macieira from comment #7) > You're not accounting for loss of performance in the shared libraries that > can't use -Bsymbolic due to the copy relocations, both at load-time > (relocations by name) and at runtime (indirect addressing to local symbols). > Maybe our experiences with shared libraries isn't the same: mine is that > libraries are 10x bigger and more complex than the applications using them. > > So I submit that your benchmarks are incomplete. The keyword here is psABI compliant. Since your shared library isn't psABI compliant, all bets are off.
(In reply to Thiago Macieira from comment #7) > (In reply to H.J. Lu from comment #4) > > (In reply to Thiago Macieira from comment #3) > > > Thanks H.J.! > > > > > > Can I ask that -fsymbolic be the default? Otherwise, code with -fPIE MUST > > > add -fsymbolic in GCC 5+, but can't add it prior because the option didn't > > > exist. Please leave that for a release or two so that we can adapt > > > > Linux kernel has things like > > > > M16_CFLAGS := $(call cc-option, -m16, $(CODE16GCC_CFLAGS)) > > > > It adds -m16 only if it exits. Why can't KDE do something like this? > > There are many ways to achieve it and it works with all compilers. > > We can. The problem is not the ability, it's the need to do it. Now, you have a need :-). > No buildsystem currently has support for -fsymbolic, since it doesn't exist > yet. However, the lack of such option will produce subtle bugs. I'm asking > for a grace period until the changes propagate. > I don't think GCC default should cater shared libraries which aren't psABI compliant. It is the risk you take when using a non-psABI compliant library.
(In reply to Thiago Macieira from comment #8) > (In reply to H.J. Lu from comment #6) > > /export/build/gnu/gcc-5/build-x86_64-linux/gcc/xgcc > > -B/export/build/gnu/gcc-5/build-x86_64-linux/gcc/ -flto -Wl,-Bsymbolic -g > > -O2 -fsymbolic -shared -o libb.so b.o a.o > > Will the LTO be required here? Or will it work without LTO too? > > > Dump of assembler code for function bar: > > 0x00000000000005e0 <+0>: movl $0x1e,0x2002be(%rip) # 0x2008a8 <a> > > 0x00000000000005ea <+10>: retq > > End of assembler dump. LTO is required only when definition and references aren't in the same file. > I've been accomplishing the same with protected visibility, but I keep > running into linker bugs, so I haven't been able to enable it by default. > Looks like the option here will be the proper solution. Protected data symbol means that it can't be pre-emptied. It doesn't mean its address won't be external. This is true for pointer to protected function. With copy relocation, address of protected data defined in the shared library may also be external. We only know that for sure at run-time: https://sourceware.org/bugzilla/show_bug.cgi?id=17711 https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65248 What you want is -fsymbolic with -Bsymbolic.
First of all, why is this filed as a GCC bug? And what is PIE specific on the issue? I mean, if -Bsymbolic doesn't cope well with like copy relocations, then the testcase doesn't work on most targets when the binary is normal executable rather than PIE.
(In reply to Jakub Jelinek from comment #12) > First of all, why is this filed as a GCC bug? And what is PIE specific on > the issue? I mean, if -Bsymbolic doesn't cope well with like copy > relocations, then > the testcase doesn't work on most targets when the binary is normal > executable rather than PIE. It is true that the testcase doesn't work with normal executable. I don't believe GCC is wrong to generate copy relocation in PIE. It is only a GCC bug that PIE worked before, it doesn't work now and there is no command option to make PIE to work.
(In reply to H.J. Lu from comment #13) > It is true that the testcase doesn't work with normal executable. I > don't believe GCC is wrong to generate copy relocation in PIE. It > is only a GCC bug that PIE worked before, it doesn't work now and > there is no command option to make PIE to work. As it doesn't work with normal executables, I'm not convinced we need such an option or that it is right to assume it should work.
(In reply to Jakub Jelinek from comment #14) > As it doesn't work with normal executables, I'm not convinced we need such > an option or that it is right to assume it should work. People are using -Bsymbolic for very good reasons and they know the risk. In KDE, there are #if !defined(QT_BOOTSTRAPPED) && defined(QT_REDUCE_RELOCATIONS) && defined(__ELF__) && !defined(__PIC__) && !defined(__PIE__) # error "You must build your code with position independent code if Qt was built with -reduce-relocations. "\ "Compile your code with -fPIC or -fPIE." #endif Unless we state -Bsymbolic should never be used with GCC, we should provide a way to make PIE to compatible with -Bsymbolic.
(In reply to H.J. Lu from comment #15) > Unless we state -Bsymbolic should never be used with GCC, we should > provide a way to make PIE to compatible with -Bsymbolic. Well, there is a way, build with -fPIC rather than -fPIE.
(In reply to Jakub Jelinek from comment #16) > (In reply to H.J. Lu from comment #15) > > Unless we state -Bsymbolic should never be used with GCC, we should > > provide a way to make PIE to compatible with -Bsymbolic. > > Well, there is a way, build with -fPIC rather than -fPIE. Since -Bsymbolic is allowed with PIE, we should take advantage of it. -fsymbolic -fPIE is the same as -fPIC and -fsymbolic -fPIC generates much better codes for shared library.
(In reply to H.J. Lu from comment #17) > Since -Bsymbolic is allowed with PIE, we should take advantage of it. > -fsymbolic -fPIE is the same as -fPIC and -fsymbolic -fPIC generates > much better codes for shared library. -fsymbolic -fPIE isn't the same as -fPIC. It is between -fPIC and -fPIE.
I know it isn't the same. But, IMHO this isn't worth adding another knob which will be a maintainance nightmare.
I've just been informed that Clang 3.6 with -fPIE -flto has the same problem, but I have not been able to reproduce. I find it unexpected that it would do that in LTO mode but not otherwise. As it stands, -Bsymbolic, -Bsymbolic-functions and protected visibility are useless functionality, so long as the symbols can be copy-relocated. Do you think this should be treated as a feature suggestion instead of a bug report?
(In reply to Jakub Jelinek from comment #19) > I know it isn't the same. But, IMHO this isn't worth adding another knob > which will be a maintainance nightmare. Performance difference between -fsymbolic -fPIE and -fPIC on PIE is about 2%. X86 and x86-64 Android don't support copy reloc. All executables are compiled as PIE. -fPIE no longer works on Android. Can you explain why -fsymbolic is a maintenance nightmare?
Created attachment 35427 [details] A patch to enable -fsymbolic in LTO if -Bsymbolic is used This patch improves LTO when -Bsymbolic is used.
$ pmap `pidof qtcreator` | perl -ne '@_ = split / +/; if ($_[6] eq "r-xp" && $_[7] !~ /\[/) { $_[1] =~ s/K//; $total += $_[1]; $bin = $_[1] unless $bin; } END { print "$bin $total\n"; }' 72 166164 That is, the size of the binary's text segment is 72k and the size of all the library's text segments is 162 MB (granted, this includes .rodata sections). My assertion is that keeping copy relocations is optimising for 0.05% of the codebase. I am asking that we begin reversing that decision. We can do it by opt-in, like Qt 5 tried to do: some large libraries, when they do their next binary incompatible release, enable the feature, causing the applications to stop doing copy relocations. I'd also like ld to refuse to link if copy relocations are required and the symbol comes from a library that used -fvisibility=protected -fsymbolic -Wl,-Bsymbolic (isn't that what DF_SYMBOLIC is for?)
(In reply to Thiago Macieira from comment #23) > $ pmap `pidof qtcreator` | perl -ne '@_ = split / +/; if ($_[6] eq "r-xp" && > $_[7] !~ /\[/) { $_[1] =~ s/K//; $total += $_[1]; $bin = $_[1] unless $bin; > } END { print "$bin $total\n"; }' > 72 166164 > > That is, the size of the binary's text segment is 72k and the size of all > the library's text segments is 162 MB (granted, this includes .rodata > sections). > > My assertion is that keeping copy relocations is optimising for 0.05% of the > codebase. Copy relocation is the part of the psABI. What you did violates the psABI, as it is incompatible with the normal executable. > I am asking that we begin reversing that decision. We can do it by opt-in, > like Qt 5 tried to do: some large libraries, when they do their next binary > incompatible release, enable the feature, causing the applications to stop > doing copy relocations. I'd also like ld to refuse to link if copy > relocations are required and the symbol comes from a library that used > -fvisibility=protected -fsymbolic -Wl,-Bsymbolic (isn't that what > DF_SYMBOLIC is for?) Link-time library != run-time library. You can't enforce run-time behavior at link-tome.
(In reply to Thiago Macieira from comment #23) > $ pmap `pidof qtcreator` | perl -ne '@_ = split / +/; if ($_[6] eq "r-xp" && > $_[7] !~ /\[/) { $_[1] =~ s/K//; $total += $_[1]; $bin = $_[1] unless $bin; > } END { print "$bin $total\n"; }' > 72 166164 > > That is, the size of the binary's text segment is 72k and the size of all > the library's text segments is 162 MB (granted, this includes .rodata > sections). > > My assertion is that keeping copy relocations is optimising for 0.05% of the > codebase. You can't judge everything from these numbers, just because KDE for some reason decides to use very small binaries. Many programs have much larger binaries, especially if they care about performance.
Plus, if KDE uses so small binaries, why don't just compile them with -fPIC then? You can then link them as normal executables or PIEs, depending on what you prefer, and still it supposedly wouldn't use copy relocations, as all references to externals would be through .got.
(In reply to Jakub Jelinek from comment #26) > Plus, if KDE uses so small binaries, why don't just compile them with -fPIC > then? > You can then link them as normal executables or PIEs, depending on what you > prefer, and still it supposedly wouldn't use copy relocations, as all > references to externals would be through .got. Can you guarantee that the linker won't generate copy relocs for -fPIC? Anyway, I know I am generalising from a sample of 1, but in my experience as a desktop developer, the mass of library code is much bigger and more complex than the application code. This is probably not true in large single-instance applications -- for example, mysqld's text segment is of comparable size to the total size of the libraries it uses. Still, if this were solved properly, relocations that resolved back into the executable itself would still be bound locally, even position-dependently if that's how the binary were built. I am asking that it stop doing copy relocations, which only applies to symbols the linker *did* detect came from elsewhere. I've also said I'll gladly tag each of those symbols with an __attribute__ so that GCC also knows it comes from elsewhere and generate the proper indirect load sequence via the GOT.
(In reply to Thiago Macieira from comment #27) > (In reply to Jakub Jelinek from comment #26) > > Plus, if KDE uses so small binaries, why don't just compile them with -fPIC > > then? > > You can then link them as normal executables or PIEs, depending on what you > > prefer, and still it supposedly wouldn't use copy relocations, as all > > references to externals would be through .got. > > Can you guarantee that the linker won't generate copy relocs for -fPIC? Yes.
(In reply to Thiago Macieira from comment #27) > Still, if this were solved properly, relocations that resolved back into the > executable itself would still be bound locally, even position-dependently if > that's how the binary were built. I am asking that it stop doing copy > relocations, which only applies to symbols the linker *did* detect came from > elsewhere. You are missing the point of copy relocations. Consider: int a = 1; extern int b, c; int foo (void) { return a + b + c; } compiled with -fno-pic or -fpie. a is known to be defined in the executable, but b and c are externals. Without copy relocations you'd need to emit significantly slower code (extra .got reference or similar) for all the accesses to the externals, with copy relocations you can optimistically assume they will likely be defined in the executable (usual case for larger programs, at least for C shared libraries people avoid exporting variables from shared libraries if easily possible), and if not, the linker will create copy relocations. Only with whole program (LTO or similar) compilation, when you can talk to the linker, you could find out if the externals from some TU are defined within the executable or not.
(In reply to H.J. Lu from comment #28) > (In reply to Thiago Macieira from comment #27) > > (In reply to Jakub Jelinek from comment #26) > > Can you guarantee that the linker won't generate copy relocs for -fPIC? > > Yes. Thanks. We'll start requiring -fPIC as of Qt 5.4.2 and we'll advise Linux distributions to backport the patch if they also upgrade GCC from 4.9. For reference: https://codereview.qt-project.org/111787 Please treat this bug report now as a feature suggestion.
(In reply to Jakub Jelinek from comment #29) > You are missing the point of copy relocations. Consider: > int a = 1; > extern int b, c; > int foo (void) > { > return a + b + c; > } > compiled with -fno-pic or -fpie. a is known to be defined in the > executable, but b and c are externals. Without copy relocations you'd need > to emit significantly slower code (extra .got reference or similar) for all > the accesses to the externals, with copy relocations you can optimistically > assume they will likely be defined in the executable (usual case for larger > programs, at least for C shared libraries people avoid exporting variables > from shared libraries if easily possible), and if not, the linker will > create copy relocations. That is true. But if you place the same code in a library, then now all accesses must be indirect, even for a. My assertion isn't about the usefulness of copy relocations, it's that they are optimising for the wrong thing. The size and complexity of libraries and plugins in desktop applications is orders of magnitude above that of the application codebases. > Only with whole program (LTO or similar) compilation, when you can talk to > the linker, you could find out if the externals from some TU are defined > within the executable or not. Or if we tag them appropriately. int a = 1; extern int b; __attribute__((dllimport)) extern int c; int foo(void) { return a + b + c; } Now the compiler knows that a is in the local executable and it can assume that b is too, but it also knows that c isn't and must be accessed indirectly. This did not require LTO. Modern libraries already all have a macro preceding all the function and variable declarations meant to be used by other DSOs, ever since Ulrich Drepper's "dso-howto" manual. The macro tags are required so that we have a proper __attribute__((visibility("default"))) when the library is compiled with -fvisibility=hidden. Moreover, the same tag expands to __declspec(dllexport) or __declspec(dllimport) on Windows if the library is cross-platform. So the precedent is there and modern libraries are mostly ready to make use of the feature.
GCC 5.2 is being released, adjusting target milestone to 5.3.
gcc 5.2.1 can no longer reproduce the testcase. Has the situation changed?
(In reply to Thiago Macieira from comment #33) > gcc 5.2.1 can no longer reproduce the testcase. > > Has the situation changed? GCC 5 r226650 still fails for me.
I've only tried with SUSE's distro build, as my own GCC build is now GCC 6. However, I can reproduce it with my GCC 6 build. Let me investigate if OpenSUSE applied a patch.
GCC 5.3 is being released, adjusting target milestone.
I think there is no bug here, so there is nothing to fix. As has been said, if binaries access vars defined in -Wl,-Bsymbolic libraries and they weren't PIE, they didn't work before either. And, you can always build the binaries with -fpic if you want to mix them with -Wl,-Bsymbolic.