Using gcc on an embedded system, with its own prefix and lib dirs.

[Lance Fredrickson]

I've made a project called 'Tomatoware' that creates a native 
development environment for tomato firmware supported embedded routers.
https://github.com/lancethepants/tomatoware
Tomato firmware uses a read-only filesystem.  Because of this, 
Tomatoware then uses external storage media connected to the USB ports 
on the device to create it's own filesystem.
This is very similar to optware for routers if you've ever used it. 
Everything is contained to the prefix Tomatoware has been compiled for.
ie
/mmc/bin/
/mmc/sbin
/mmc/lib/
/mmc/include
/mmc/etc

The toolchain used to build Tomatoware was built using gcc 4.6.4, so I 
am also using 4.6.4 for Tomatoware.

Like everything in Tomatoware, gcc has been compiled with --prefix=/mmc 
(also other prefixes).  So far most things have worked well when 
compiling on the router using Tomatoware.
I've discovered though by running autoconf configure scripts, gcc wants 
to use libraries found in /lib and /usr/lib

configure:12025: gcc -o conftest -g -O2 -pthread conftest.c >&5
/lib/libdl.so.0: undefined reference to `_dl_lookup_hash'

Tomatoware uses a newer version of uclibc then what tomato is using. 
That explains the failure, but I don't want it to even know about /lib 
/usr/lib directories.
Detecting system threading support in particular is one place where 
things tend to fail.
Tomato firmware doesn't have an /include directory, and does not contain 
header files at all, so that has not been an issue.

I've tried passing LDFLAGS commands to configure to supersede what gcc 
wants to do.

configure:12025: gcc -o conftest -g -O2 -pthread  -L/mmc/lib 
conftest.c   >&5
/lib/libdl.so.0: undefined reference to `_dl_lookup_hash'

I'm guessing since the LDFLAGS is coming after -pthread in the configure 
script, that is why it isn't working.  I imagine many other autoconf 
configure scripts work the same.

Looking around I read that the following (which formats things nicely) 
that shows the lib search path of gcc.
'gcc -print-search-dirs | sed '/^lib/b 1;d;:1;s,/[^/.][^/]*/\.\./,/,;t 
1;s,:[^=]*=,:;,;s,;,;  ,g' | tr \; \\012'n'
/mmc/lib/gcc/mipsel-linux/4.6.4/:/mmc/mipsel-linux/lib/mipsel-linux/4.6.4/:/mmc/mipsel-linux/lib/:/mmc/lib/mipsel-linux/4.6.4/:/mmc/lib/:/lib/mipsel-linux/4.6.4/:/lib/:/usr/lib/mipsel-linux/4.6.4/:/usr/lib/

It shows that /mmc/lib is in there, and it is coming before /lib and 
/usr/lib
This however doesn't seem to be the case when linking.  It should be 
finding libdl in /mmc/lib before /lib and /usr/lib, but it isn't

I've also tried setting the LIBRARY_PATH environment variable to 
/mmc/lib.  Using the same gcc -print-search-dirs command I can confirm 
that it is being inserted in to the library path.
It was already present, but now is in there twice, before /lib and 
/usr/lib, but that doesn't help.
The gcc documentation says this should work "When configured as a native 
compiler..."  Tomatoware and gcc has been cross-compiled to be a "native 
compiler" for the embedded system.  Not sure if this qualifies as a 
"native compiler"  by the documentation description though.

I've also read it looks at /etc/ld.so.conf, and that seems to be the 
case. gcc is looking at that, and ignoring everything else. 
/etc/ld.so.conf is part of the read-only file system. I can override it 
though with 'mount --bind /mmc/etc/ld.so.conf /etc/ld.so.conf'
My file includes /mmc/lib inserted at the top, and then gcc works as it 
should.

This is a solution, but one I don't really like. It adds complexity for 
setting this up for users, and I feel like I should be able to get gcc 
to actually use the library path it states in 'gcc -print-search-dirs'.
My guess is that since I'm cross-compiling gcc, that is causing it to 
behave differently than what I find documented. Is this correct?
Is it possible and plausible to get gcc for this setup to use its 
library search path, and possibly even remove all references to /lib/ 
/usr/lib.  Tomatoware should not even be aware of the embedded file 
system and should work completely independently of it.

If you made it this far.
thanks
-Lance