Library search path problem on x86_64 machines (lib64 vs lib)

Tue Nov 30 13:12:00 GMT 2010

I've found a library search path problem with GCC 4.3.1. I wonder
whether this is a bug in GCC or in Debian's symbolic link settings
and what the current status is (with new GCC versions).

The summary:

GCC searches all .../lib64 directories, then all .../lib directories,
but Debian has a symbolic link /usr/lib64 -> /usr/lib, which means
that /usr/lib will have the precedence over all the other .../lib
directories. This is wrong!

The problem is particularly important because in my automatic
installation of GMP and MPFR, I got no errors, but incorrect
versions of GMP and MPFR (possibly more buggy) were used. I've
just noticed it today, several weeks after the installation.

Debian's GCC 4.3.2 has a different behavior, but I don't know whether
there has been a change or Debian did anything special.

I give the details of all my tests (reordered to give the simplest
tests first) below. First note that this is a Debian/stable machine,
with two GCC versions installed: 4.3.1 (by the system administrator)
and 4.3.2 (from Debian). As GCC 4.3.2 doesn't compile GMP correctly,
I had to use 4.3.1 for GMP and MPFR.

patate% cat /etc/debian_version
5.0.7
patate% which gcc-4.3
/usr/bin/gcc-4.3
patate% which gcc-4.3.1
/usr/local/bin/gcc-4.3.1

patate% export C_INCLUDE_PATH=/home/vlefevre/gmp/athlon64/include:/home/vlefevre/x86_64/include
patate% export LIBRARY_PATH=/home/vlefevre/gmp/athlon64/lib:/home/vlefevre/x86_64/lib
patate% export LD_LIBRARY_PATH=$LIBRARY_PATH

patate% echo 'main() { return 0; }' > tst.c
patate% gcc-4.3 tst.c -o tst -lgmp
patate% ldd tst
        linux-vdso.so.1 =>  (0x00007fff05dff000)
        libgmp.so.10 => /home/vlefevre/gmp/athlon64/lib/libgmp.so.10 (0x00007f3fea80e000)
        libc.so.6 => /lib/libc.so.6 (0x00007f3fea4bb000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3feaa7c000)
patate% gcc-4.3.1 tst.c -o tst -lgmp
patate% ldd tst
        linux-vdso.so.1 =>  (0x00007fffce981000)
        libgmp.so.3 => /home/vlefevre/x86_64/lib/libgmp.so.3 (0x00007fedc2a97000)
        libc.so.6 => /lib/libc.so.6 (0x00007fedc266a000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fedc29bd000)

Why "libgmp.so.3 => /home/vlefevre/x86_64/lib/libgmp.so.3"?

My explanation: gcc-4.3.1 linked tst with libgmp.so.3, found in /usr/lib
(see below). But since the path isn't recorded by ELF, it is chosen from
$LD_LIBRARY_PATH at run time (or by ldd).
  * /home/vlefevre/gmp/athlon64/lib contains libgmp.so.10, which is not
    compatible.
  * /home/vlefevre/x86_64/lib contains libgmp.so.3, which is chosen.

patate% gcc-4.3.1 -v
Using built-in specs.
Target: x86_64-unknown-linux-gnu
Configured with: ../gcc-4.3.1/configure --prefix=/usr/local/stow/gcc-4.3.1 --program-suffix=-4.3.1 --enable-threads --enable-tls
Thread model: posix
gcc version 4.3.1 (GCC)

patate% gcc-4.3 -print-file-name=libgmp.so
/home/vlefevre/gmp/athlon64/lib/../lib/libgmp.so

patate% gcc-4.3.1 -print-file-name=libgmp.so
/usr/lib/../lib64/libgmp.so

patate% gcc-4.3 -print-search-dirs
install: /usr/lib/gcc/x86_64-linux-gnu/4.3.2/
programs: =/usr/lib/gcc/x86_64-linux-gnu/4.3.2/:/usr/lib/gcc/x86_64-linux-gnu/4.3.2/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.3.2/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/libexec/gcc/x86_64-linux-gnu/4.3.2/:/usr/libexec/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.3.2/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../x86_64-linux-gnu/bin/x86_64-linux-gnu/4.3.2/:/usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../x86_64-linux-gnu/bin/
libraries: =/home/vlefevre/gmp/athlon64/lib/x86_64-linux-gnu/4.3.2/:/home/vlefevre/gmp/athlon64/lib/../lib/:/home/vlefevre/x86_64/lib/x86_64-linux-gnu/4.3.2/:/home/vlefevre/x86_64/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.3.2/:/usr/lib/gcc/x86_64-linux-gnu/4.3.2/:/usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/4.3.2/:/usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../x86_64-linux-gnu/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../x86_64-linux-gnu/4.3.2/:/usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../lib/:/lib/x86_64-linux-gnu/4.3.2/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/4.3.2/:/usr/lib/../lib/:/home/vlefevre/gmp/athlon64/lib/:/home/vlefevre/x86_64/lib/:/usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../../x86_64-linux-gnu/lib/:/usr/lib/gcc/x86_64-linux-gnu/4.3.2/../../../:/lib/:/usr/lib/

patate% gcc-4.3.1 -print-search-dirs
install: /usr/local/stow/gcc-4.3.1/lib/gcc/x86_64-unknown-linux-gnu/4.3.1/
programs: =/usr/local/stow/gcc-4.3.1/libexec/gcc/x86_64-unknown-linux-gnu/4.3.1/:/usr/local/stow/gcc-4.3.1/libexec/gcc/x86_64-unknown-linux-gnu/4.3.1/:/usr/local/stow/gcc-4.3.1/libexec/gcc/x86_64-unknown-linux-gnu/:/usr/local/stow/gcc-4.3.1/lib/gcc/x86_64-unknown-linux-gnu/4.3.1/:/usr/local/stow/gcc-4.3.1/lib/gcc/x86_64-unknown-linux-gnu/:/usr/libexec/gcc/x86_64-unknown-linux-gnu/4.3.1/:/usr/libexec/gcc/x86_64-unknown-linux-gnu/:/usr/lib/gcc/x86_64-unknown-linux-gnu/4.3.1/:/usr/lib/gcc/x86_64-unknown-linux-gnu/:/usr/local/stow/gcc-4.3.1/lib/gcc/x86_64-unknown-linux-gnu/4.3.1/../../../../x86_64-unknown-linux-gnu/bin/x86_64-unknown-linux-gnu/4.3.1/:/usr/local/stow/gcc-4.3.1/lib/gcc/x86_64-unknown-linux-gnu/4.3.1/../../../../x86_64-unknown-linux-gnu/bin/
libraries: =/home/vlefevre/gmp/athlon64/lib/x86_64-unknown-linux-gnu/4.3.1/:/home/vlefevre/gmp/athlon64/lib/../lib64/:/home/vlefevre/x86_64/lib/x86_64-unknown-linux-gnu/4.3.1/:/home/vlefevre/x86_64/lib/../lib64/:/usr/local/stow/gcc-4.3.1/lib/gcc/x86_64-unknown-linux-gnu/4.3.1/:/usr/lib/gcc/x86_64-unknown-linux-gnu/4.3.1/:/usr/local/stow/gcc-4.3.1/lib/gcc/x86_64-unknown-linux-gnu/4.3.1/../../../../x86_64-unknown-linux-gnu/lib/x86_64-unknown-linux-gnu/4.3.1/:/usr/local/stow/gcc-4.3.1/lib/gcc/x86_64-unknown-linux-gnu/4.3.1/../../../../x86_64-unknown-linux-gnu/lib/../lib64/:/usr/local/stow/gcc-4.3.1/lib/gcc/x86_64-unknown-linux-gnu/4.3.1/../../../x86_64-unknown-linux-gnu/4.3.1/:/usr/local/stow/gcc-4.3.1/lib/gcc/x86_64-unknown-linux-gnu/4.3.1/../../../../lib64/:/lib/x86_64-unknown-linux-gnu/4.3.1/:/lib/../lib64/:/usr/lib/x86_64-unknown-linux-gnu/4.3.1/:/usr/lib/../lib64/:/home/vlefevre/gmp/athlon64/lib/:/home/vlefevre/x86_64/lib/:/usr/local/stow/gcc-4.3.1/lib/gcc/x86_64-unknown-linux-gnu/4.3.1/../../../../x86_64-unknown-linux-gnu/lib/:/usr/local/stow/gcc-4.3.1/lib/gcc/x86_64-unknown-linux-gnu/4.3.1/../../../:/lib/:/usr/lib/

patate% ls -l /usr/lib64
lrwxrwxrwx 1 root root 3 2009-04-27 14:12:46 /usr/lib64 -> lib

GCC searches all .../lib64 directories, then all .../lib directories,
but Debian has a symbolic link /usr/lib64 -> /usr/lib, which means
that /usr/lib will have the precedence over all the other .../lib
directories. This is wrong!

patate% rm -rf mpfr-3.0.0
patate% tar xf mpfr-3.0.0.tar.bz2
patate% cd mpfr-3.0.0
patate% patch -N -Z -p1 < ~/wd/mpfr/web/mpfr-3.0.0/allpatches
[patch output]
patate% ./configure --prefix=$HOME/tmp/mpfr >& configure.out
patate% make -j16 > make.out
patate% make -j16 check > check.out
patate% make install > install.out
patate% ldd ~/tmp/mpfr/lib/libmpfr.so.4.0.0
        linux-vdso.so.1 =>  (0x00007fff599ff000)
        libgmp.so.3 => /home/vlefevre/x86_64/lib/libgmp.so.3 (0x00007fee40b7f000)
        libc.so.6 => /lib/libc.so.6 (0x00007fee407ff000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fee40f13000)

configure.out contains:

checking if gmp.h version and libgmp version are the same... (5.0.1/4.2.4) no
configure: WARNING: 'gmp.h' and 'libgmp' seems to have different versions or
configure: WARNING: we cannot run a program linked with GMP (if you cannot
configure: WARNING: see the version numbers above). A cause may be different
configure: WARNING: GMP versions with different ABI's.
configure: WARNING: However since we can't use 'libtool' inside the configure,
configure: WARNING: we can't be sure. See 'config.log' for details.

This is due to the fact that /home/vlefevre/gmp/athlon64/include/gmp.h comes
from GMP 5.0.1, but /home/vlefevre/x86_64/lib/libgmp.so is GMP 4.2.4 (see the
above test on tst.c).

libmpfr.la contains a third path for GMP: /usr/lib64/libgmp.la !!!
Indeed, in make.out, we can see that libtool has replaced -lgmp by
/usr/lib/libgmp.so (/usr/lib64 is a symbolic link to /usr/lib).
And libtool used this version.

Thus check.out contains:

WARNING! The versions of gmp.h (5.0.1) and libgmp (4.2.4) do not match.
This may lead to errors, in particular with MPFR. If some tests fail,
please check that first. As we are not sure, we do not regard this as
an error.

But this automatic installations, such warnings are not detected.

patate% LIBRARY_PATH=/home/vlefevre/tmp/mpfr/lib:$LIBRARY_PATH
patate% LD_LIBRARY_PATH=$LIBRARY_PATH
patate% env | grep LIBRARY_PATH
LIBRARY_PATH=/home/vlefevre/tmp/mpfr/lib:/home/vlefevre/gmp/athlon64/lib:/home/vlefevre/x86_64/lib
LD_LIBRARY_PATH=/home/vlefevre/tmp/mpfr/lib:/home/vlefevre/gmp/athlon64/lib:/home/vlefevre/x86_64/lib
patate% cd ..

patate% gcc-4.3 tst.c -o tst -lmpfr -lgmp
/usr/bin/ld: warning: libgmp.so.3, needed by /home/vlefevre/tmp/mpfr/lib/../lib/libmpfr.so, may conflict with libgmp.so.10
patate% ldd tst
        linux-vdso.so.1 =>  (0x00007fff04ea0000)
        libmpfr.so.4 => /home/vlefevre/tmp/mpfr/lib/libmpfr.so.4 (0x00007fc02558c000)
        libgmp.so.10 => /home/vlefevre/gmp/athlon64/lib/libgmp.so.10 (0x00007fc02531e000)
        libc.so.6 => /lib/libc.so.6 (0x00007fc024fcb000)
        libgmp.so.3 => /home/vlefevre/x86_64/lib/libgmp.so.3 (0x00007fc02588b000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fc0257df000)
patate% gcc-4.3.1 tst.c -o tst -lmpfr -lgmp
patate% ldd tst
        linux-vdso.so.1 =>  (0x00007fffb13ff000)
        libmpfr.so.1 => /home/vlefevre/x86_64/lib/libmpfr.so.1 (0x00007f8d4dc22000)
        libgmp.so.3 => /home/vlefevre/x86_64/lib/libgmp.so.3 (0x00007f8d4da19000)
        libc.so.6 => /lib/libc.so.6 (0x00007f8d4d6c6000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8d4db57000)

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <http://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <http://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / Arénaire project (LIP, ENS-Lyon)