porting libstdc++ over a network simulator

[Mathieu Lacage <mathieu dot lacage at sophia dot inria dot fr>]

hi,

I hope this ML is the right forum for this kind of question. If not, I
would really appreciate being redirected to a more appropriate location.

In the context of ns-3 (http://www.nsnam.org), I have been working on
adding a layer to ns-3 which allows me to load multiple times an elf
binary in memory. What I do is trick my elf loader (the glibc elf
loader) to load multiple times the same binary in memory. Then, I have
to provide a valid runtime environment to the binary by dynamically
replacing the libc (the result is pretty similar to valgrind, even
though the way it is done is radically different). The former seems to
work just fine. The latter is still problematic, hence, this email.

So far, I have been adding more functionality to my trimmed-down libc
when the higher-layers needed it and it seems to work just fine.
However, things started to go downhill when I tried to use std::cout
from one of my simulated programs. The error I get is shown below:

#0  _IO_fwrite (buf=0x2ab259a12dc6, size=1, count=5, fp=0x0) at iofwrite.c:43
#1  0x00002ab253599c77 in ns3::ProcessManager::Fwrite (this=0x62c350, ptr=0x2ab259a12dc6, size=1, nmemb=5, stream=0x0)
    at ../src/process/process-manager.cc:1554
#2  0x00002ab2535aa933 in simu_fwrite (ptr=0x2ab259a12dc6, size=1, nmemb=5, stream=0x0) at ../src/process/simu.cc:218
#3  0x00002ab2598104e3 in fwrite (ptr=0x2ab259a12dc6, size=1, nmemb=5, stream=0x0) at ../src/process/libc-fn.c:230
#4  0x00002ab259cb9c24 in __gnu_cxx::stdio_sync_filebuf<char, std::char_traits<char> >::xsputn (this=0x2ab259f2c360, 
    __s=0x2ab259a12dc6 "send ", __n=5)
    at /build/buildd/gcc-4.2-4.2.1/build/x86_64-linux-gnu/libstdc++-v3/include/ext/stdio_sync_filebuf.h:217
#5  0x00002ab259cc4c00 in std::basic_streambuf<char, std::char_traits<char> >::sputn (this=0x2ab259f2c360, 
    __s=0x2ab259a12dc6 "send ", __n=5)
    at /build/buildd/gcc-4.2-4.2.1/build/x86_64-linux-gnu/libstdc++-v3/include/streambuf:447
#6  0x00002ab259cbadcb in std::__ostream_write<char, std::char_traits<char> > (__out=@0x2ab259f2bb80, 
    __s=0x2ab259a12dc6 "send ", __n=5)
    at /build/buildd/gcc-4.2-4.2.1/build/x86_64-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:52
#7  0x00002ab259cbe987 in std::__ostream_insert<char, std::char_traits<char> > (__out=@0x2ab259f2bb80, 
    __s=0x2ab259a12dc6 "send ", __n=5)
    at /build/buildd/gcc-4.2-4.2.1/build/x86_64-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:103
#8  0x00002ab259cbea7e in std::operator<< <std::char_traits<char> > (__out=@0x2ab259f2bb80, __s=0x2ab259a12dc6 "send ")
    at /build/buildd/gcc-4.2-4.2.1/build/x86_64-linux-gnu/libstdc++-v3/include/ostream:517
#9  0x00002ab259a12d33 in main (argc=2, argv=0x62eeb0) at ../examples/process-udp-client.cc:34
#10 0x00002ab2535a0b27 in ns3::ProcessManager::CreateWithLoader (argc=2, argv=0x62eeb0)
    at ../src/process/process-manager.cc:162
#11 0x00002ab2535a40a9 in ns3::ProcessManager::DoCreateThread () at ../src/process/process-manager.cc:447
#12 0x00002ab2577ef040 in ?? () from /usr/lib/debug/libc.so.6
#13 0x0000000000000000 in ?? ()

What this shows is that, when I write:
std::cout << "send " << n << " bytes" << std::endl;
libstdc++ ends up calling the underlying libc's fwrite function from
__gnu_cxx::stdio_sync_filebuf which then calls into my own fwrite
implementation. So far, so good. The issue, though, is that my ::fwrite
implementation receives a NULL pointer as a FILE *, hence, causing my
program to crash when it tries to access it. The question, then, is
why ?

I spent quite a bit of time looking at the stdc++ source code and
eventually found src/ios_init.cc which seems to be responsible for
constructing std::cout. I believe that ios_base::Init::Init is indeed
called before I can reach my own code (I have done some extensive
debugging to catch calls to that function) so, I would expect std::cout
to be correctly initialized.

I tried to gather more information about my system's libc/libstdc++.
Here is what I have:

  1) the implementation used by my system's
__gnu_cxx::__exchange_and_add_dispatch seems to use
pthread_mutex_lock/unlock (I have an implementation of these too).

  2) gdb shows me that the FILE * field of stdio_sync_filebuf (_M_file)
is non-null
(gdb) p ('__gnu_cxx::stdio_sync_filebuf<char, std::char_traits<char> >' *) std::cout._M_streambuf
$7 = (__gnu_cxx::stdio_sync_filebuf<char,std::char_traits<char> > *) 0x2ab257309360
(gdb) p $7->_M_file
$9 = (__c_file * const) 0x2ab257b01780
(gdb) p stdout
$10 = (struct _IO_FILE *) 0x2ab257b01780
(gdb) 

2) above would hint at something which is not an initialization problem
but I have to confess that I do not trust the value reported by gdb
because I have been unable to figure out which of the std::cout
instances it is printing.

So, at this point, I don't really know in which direction I should go
other than try to figure out which std::cout gdb is printing, and, then,
proceed from there: I suspect that a good night of sleep will help when
I get back to this tomorrow but, it might be that someone how knows the
details of libstdc++ knows how libstdc++ could pass a NULL pointer
to ::fwrite and that would probably help me quite a bit.

regards,
Mathieu