Re: porting libstdc++ over a network simulator

[Mathieu Lacage <mathieu dot lacage at sophia dot inria dot fr>]

Since I did not get any hate email saying that this was totally
inappropriate for this ML, for the record, the bug was coming from a
missing RTLD_DEEPBIND which was forcing making my code to initialize the
wrong version of stdout, hence, leaving the stdout I cared about still
initialized to 0, hence, making std::cout use the 0-initialized version
of stdout.

Anyhow, this works (on the platforms I tested it with) and it is pretty
cool. You can check it out:

hg clone http://code.nsnam.org/mathieu/ns-3-simu
cd ns-3-simu
./waf
./waf --shell
./build/debug/examples/process

Output:
send 1001 bytes
got bytes n=1001

Which shows that multiple instances of libstdc++ are virtualized within
this process.

Thanks again for this cool piece of software and the great (internal)
porting documentation !

regards,
Mathieu

On Mon, 2008-08-04 at 18:18 -0700, Mathieu Lacage wrote:
> hi,
> 
> I hope this ML is the right forum for this kind of question. If not, I
> would really appreciate being redirected to a more appropriate location.
> 
> In the context of ns-3 (http://www.nsnam.org), I have been working on
> adding a layer to ns-3 which allows me to load multiple times an elf
> binary in memory. What I do is trick my elf loader (the glibc elf
> loader) to load multiple times the same binary in memory. Then, I have
> to provide a valid runtime environment to the binary by dynamically
> replacing the libc (the result is pretty similar to valgrind, even
> though the way it is done is radically different). The former seems to
> work just fine. The latter is still problematic, hence, this email.
> 
> So far, I have been adding more functionality to my trimmed-down libc
> when the higher-layers needed it and it seems to work just fine.
> However, things started to go downhill when I tried to use std::cout
> from one of my simulated programs. The error I get is shown below:
> 
> #0  _IO_fwrite (buf=0x2ab259a12dc6, size=1, count=5, fp=0x0) at iofwrite.c:43
> #1  0x00002ab253599c77 in ns3::ProcessManager::Fwrite (this=0x62c350, ptr=0x2ab259a12dc6, size=1, nmemb=5, stream=0x0)
>     at ../src/process/process-manager.cc:1554
> #2  0x00002ab2535aa933 in simu_fwrite (ptr=0x2ab259a12dc6, size=1, nmemb=5, stream=0x0) at ../src/process/simu.cc:218
> #3  0x00002ab2598104e3 in fwrite (ptr=0x2ab259a12dc6, size=1, nmemb=5, stream=0x0) at ../src/process/libc-fn.c:230
> #4  0x00002ab259cb9c24 in __gnu_cxx::stdio_sync_filebuf<char, std::char_traits<char> >::xsputn (this=0x2ab259f2c360, 
>     __s=0x2ab259a12dc6 "send ", __n=5)
>     at /build/buildd/gcc-4.2-4.2.1/build/x86_64-linux-gnu/libstdc++-v3/include/ext/stdio_sync_filebuf.h:217
> #5  0x00002ab259cc4c00 in std::basic_streambuf<char, std::char_traits<char> >::sputn (this=0x2ab259f2c360, 
>     __s=0x2ab259a12dc6 "send ", __n=5)
>     at /build/buildd/gcc-4.2-4.2.1/build/x86_64-linux-gnu/libstdc++-v3/include/streambuf:447
> #6  0x00002ab259cbadcb in std::__ostream_write<char, std::char_traits<char> > (__out=@0x2ab259f2bb80, 
>     __s=0x2ab259a12dc6 "send ", __n=5)
>     at /build/buildd/gcc-4.2-4.2.1/build/x86_64-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:52
> #7  0x00002ab259cbe987 in std::__ostream_insert<char, std::char_traits<char> > (__out=@0x2ab259f2bb80, 
>     __s=0x2ab259a12dc6 "send ", __n=5)
>     at /build/buildd/gcc-4.2-4.2.1/build/x86_64-linux-gnu/libstdc++-v3/include/bits/ostream_insert.h:103
> #8  0x00002ab259cbea7e in std::operator<< <std::char_traits<char> > (__out=@0x2ab259f2bb80, __s=0x2ab259a12dc6 "send ")
>     at /build/buildd/gcc-4.2-4.2.1/build/x86_64-linux-gnu/libstdc++-v3/include/ostream:517
> #9  0x00002ab259a12d33 in main (argc=2, argv=0x62eeb0) at ../examples/process-udp-client.cc:34
> #10 0x00002ab2535a0b27 in ns3::ProcessManager::CreateWithLoader (argc=2, argv=0x62eeb0)
>     at ../src/process/process-manager.cc:162
> #11 0x00002ab2535a40a9 in ns3::ProcessManager::DoCreateThread () at ../src/process/process-manager.cc:447
> #12 0x00002ab2577ef040 in ?? () from /usr/lib/debug/libc.so.6
> #13 0x0000000000000000 in ?? ()
> 
> What this shows is that, when I write:
> std::cout << "send " << n << " bytes" << std::endl;
> libstdc++ ends up calling the underlying libc's fwrite function from
> __gnu_cxx::stdio_sync_filebuf which then calls into my own fwrite
> implementation. So far, so good. The issue, though, is that my ::fwrite
> implementation receives a NULL pointer as a FILE *, hence, causing my
> program to crash when it tries to access it. The question, then, is
> why ?
> 
> I spent quite a bit of time looking at the stdc++ source code and
> eventually found src/ios_init.cc which seems to be responsible for
> constructing std::cout. I believe that ios_base::Init::Init is indeed
> called before I can reach my own code (I have done some extensive
> debugging to catch calls to that function) so, I would expect std::cout
> to be correctly initialized.
> 
> I tried to gather more information about my system's libc/libstdc++.
> Here is what I have:
> 
>   1) the implementation used by my system's
> __gnu_cxx::__exchange_and_add_dispatch seems to use
> pthread_mutex_lock/unlock (I have an implementation of these too).
> 
>   2) gdb shows me that the FILE * field of stdio_sync_filebuf (_M_file)
> is non-null
> (gdb) p ('__gnu_cxx::stdio_sync_filebuf<char, std::char_traits<char> >' *) std::cout._M_streambuf
> $7 = (__gnu_cxx::stdio_sync_filebuf<char,std::char_traits<char> > *) 0x2ab257309360
> (gdb) p $7->_M_file
> $9 = (__c_file * const) 0x2ab257b01780
> (gdb) p stdout
> $10 = (struct _IO_FILE *) 0x2ab257b01780
> (gdb) 
> 
> 2) above would hint at something which is not an initialization problem
> but I have to confess that I do not trust the value reported by gdb
> because I have been unable to figure out which of the std::cout
> instances it is printing.
> 
> So, at this point, I don't really know in which direction I should go
> other than try to figure out which std::cout gdb is printing, and, then,
> proceed from there: I suspect that a good night of sleep will help when
> I get back to this tomorrow but, it might be that someone how knows the
> details of libstdc++ knows how libstdc++ could pass a NULL pointer
> to ::fwrite and that would probably help me quite a bit.
> 
> regards,
> Mathieu
>