libstdc++ debugging, was:Re: Heads up: Several thousand g++ and libstdc++test failures on sparc[64]

[Andreas Tobler <toa at pop dot agri dot ch>]

Ok, I did some debugging on this topic. But I don't know if I do it 
correct. Could anyone please tell me if it is the right way?
I attach my dbg log.
At the moment I'm stuck since I don't want to follow the wrong 
direction, if.
Any corrections, confirmations are appreciated.

Thanks,
Andreas

Debugging Notes on how to map unknown symbols on solaris
--------------------------------------------------------
[enterprise:objdir/sparc-sun-solaris2.9/libstdc++-v3] andreast% ./header_cassert.exe 
Segmentation fault (core dumped)

This GDB was configured as "sparcv9-sun-solaris2.9"...
(gdb) r
Starting program: /opt/gcc/gcc-cvs/objdir/sparc-sun-solaris2.9/libstdc++-v3/header_cassert.exe 

Program received signal SIGSEGV, Segmentation fault.
0xff2cc4c0 in ?? () from /usr/local/lib/libstdc++.so.6
(gdb) bt
#0  0xff2cc4c0 in ?? () from /usr/local/lib/libstdc++.so.6
#1  0xff2e36ec in ?? () from /usr/local/lib/libstdc++.so.6
#2  0xff2e171c in ?? () from /usr/local/lib/libstdc++.so.6
#3  0xff2cc3e8 in ?? () from /usr/local/lib/libstdc++.so.6
#4  0xff2cb684 in ?? () from /usr/local/lib/libstdc++.so.6
#5  0xff3ccc28 in ?? ()
#6  0xff3cc3a4 in ?? ()
#7  0xff3d7fe8 in ?? ()
#8  0xff3c2be8 in ?? ()
(gdb) 

We want to find out what's behind 0xff2cc4c0:

(gdb) disass  0xff2cc4c0-36  0xff2cc4c0+36
Dump of assembler code from 0xff2cc49c to 0xff2cc4e4:
0xff2cc49c:     sethi  %hi(0x74400), %l7
0xff2cc4a0:     call  0xff2cb850
0xff2cc4a4:     
    add  %l7, 0x1e0, %l7        ! 0x745e0 <_ZNKSt7collateIwE10do_compareEPKwS2_S2_S2_+433>
0xff2cc4a8:     or  %g1, 0x10c, %g1
0xff2cc4ac:     ld  [ %l7 + %g1 ], %i5
0xff2cc4b0:     ldstub  [ %i5 ], %i4
0xff2cc4b4:     cmp  %i4, 0
0xff2cc4b8:     bne  0xff2cc4b0
0xff2cc4bc:     nop 
0xff2cc4c0:     ld  [ %i0 ], %g1
0xff2cc4c4:     add  %g1, %i1, %g1
0xff2cc4c8:     st  %g1, [ %i0 ]
0xff2cc4cc:     clrb  [ %i5 ]
0xff2cc4d0:     ret 
0xff2cc4d4:     restore 
0xff2cc4d8:     save  %sp, -112, %sp
0xff2cc4dc:     sethi  %hi(0x400), %g1
0xff2cc4e0:     sethi  %hi(0x74400), %l7
End of assembler dump.
(gdb) 

First we need to know where the libstdc++ is mapped in the application:

For this we take the pstack/pmap utils on solaris (man pstack/pmap):

We can't run pmap on a process which is under control of gdb, so we take the 
core file produced above. (maybe with the -F switch ?)

[enterprise:objdir/sparc-sun-solaris2.9/libstdc++-v3] andreast% pmap core 
core 'core' of 14087:   ./header_cassert.exe
00010000       8K r-x--  /opt/gcc/gcc-cvs/objdir/sparc-sun-solaris2.9/libstdc++-v3/header_cassert.exe
00020000       8K rwx--  /opt/gcc/gcc-cvs/objdir/sparc-sun-solaris2.9/libstdc++-v3/header_cassert.exe
FF160000      16K r-x--  /usr/platform/sun4u/lib/libc_psr.so.1
FF180000     688K r-x--  /usr/lib/libc.so.1
FF23C000      24K rwx--  /usr/lib/libc.so.1
FF242000       8K rwx--  /usr/lib/libc.so.1
FF250000       8K rwx-- 
FF260000      40K r-x--  /opt/gcc/gcc-cvs/objdir/gcc/libgcc_s.so.1
FF278000       8K rwx--  /opt/gcc/gcc-cvs/objdir/gcc/libgcc_s.so.1
FF280000     712K r-x--  /opt/gcc/gcc-cvs/objdir/sparc-sun-solaris2.9/libstdc++-v3/src/.libs/libstdc++.so.6.0.0
FF340000     136K rwx--  /opt/gcc/gcc-cvs/objdir/sparc-sun-solaris2.9/libstdc++-v3/src/.libs/libstdc++.so.6.0.0
FF370000      96K r-x--  /usr/lib/libm.so.1
FF396000       8K rwx--  /usr/lib/libm.so.1
FF3A0000       8K r-x--  /usr/lib/libdl.so.1
FF3B0000       8K r---- 
FF3C0000     160K r-x--  /usr/lib/ld.so.1
FF3F6000      16K rwx--  /usr/lib/ld.so.1
FFBFC000      16K rwx--    [ stack ]
 total      1968K

Here we see that the libstdc++.so.6.0.0 is loaded at 0xFF280000, 
our application breaks at 0xff2cc4c0.
To find the offending code we calculate the difference:

0xff2cc4c0 - 0xFF280000 = 0x4c4c0

Now we disassemble the libstdc++.so.6.0.0 file to find the offset 0x4c4c0:
(dis libstdc++.so.6.0.0 > libstdc++.diss)

It's in the section:

section .text._Z12__atomic_addPVii
_Z12__atomic_addPVii()

_Z12__atomic_addPVii()
        4c494:  9d e3 bf 90        save         %sp, -0x70, %sp
        4c498:  03 00 00 01        sethi        %hi(0x400), %g1
        4c49c:  2f 00 01 d1        sethi        %hi(0x74400), %l7
        4c4a0:  7f ff fc ec        call         0x4b850
        4c4a4:  ae 05 e1 e0        add          %l7, 0x1e0, %l7
        4c4a8:  82 10 61 0c        or           %g1, 0x10c, %g1
        4c4ac:  fa 05 c0 01        ld           [%l7 + %g1], %i5
        4c4b0:  f8 6f 40 00        ldstub       [%i5], %i4
        4c4b4:  80 a7 20 00        cmp          %i4, 0x0
        4c4b8:  12 bf ff fe        bne          0x4c4b0
        4c4bc:  01 00 00 00        nop
        4c4c0:  c2 06 00 00        ld           [%i0], %g1
        4c4c4:  82 00 40 19        add          %g1, %i1, %g1
        4c4c8:  c2 26 00 00        st           %g1, [%i0]
        4c4cc:  c0 2f 40 00        clrb         [%i5]
        4c4d0:  81 c7 e0 08        ret
        4c4d4:  81 e8 00 00        restore


Hm, I see _Z12__atomic_addPVii() several times in the disassembled libstdc++.
Strange!

Seven times in total. Why?

__atomic_add is defined in:

include/sparc-sun-solaris2.9/bits/atomicity.h
here for 32 bit:

static void
__attribute__ ((__unused__))
__atomic_add (volatile _Atomic_word* __mem, int __val)
{
  _Atomic_word __tmp;

  __asm__ __volatile__("1:	ldstub	[%1], %0\n\t"
		       "	cmp	%0, 0\n\t"
		       "	bne	1b\n\t"
		       "	 nop"
		       : "=&r" (__tmp)
		       : "r" (&__Atomicity_lock<0>::_S_atomicity_lock)
		       : "memory");
  *__mem += __val;
   __asm__ __volatile__("stb	%%g0, [%0]"
		       : /* no outputs */
		       : "r" (&__Atomicity_lock<0>::_S_atomicity_lock)
		       : "memory");
}

pstack core:

[enterprise:objdir/sparc-sun-solaris2.9/libstdc++-v3] andreast% pstack core
core 'core' of 14087:   ./header_cassert.exe
 ff2cc4c0 _Z12__atomic_addPVii (0, 1, 0, ff35b3d4, 0, 20cb0) + 2c
 ff2e36e4 _ZNSt8ios_base4InitC1Ev (ff35ba54, 0, 0, 0, 0, 400) + 84
 ff2e1714 _Z41__static_initialization_and_destruction_0ii (1, 1, 0, 1, 0, 1) + 60
 ff2cc3e0 ???????? (ff34ff10, 1, 2d738, 2dcb8, ff3d0800, 0)
 ff2cb67c _init    (0, 0, ff3f7344, 2dcb8, ff3ccc18, ff3f7844) + c
 ff3ccc20 call_init (ff250ed0, 0, ff250ec0, ffdfffff, 400000, 80000) + 1c0
 ff3cc39c setup    (ff3f71d0, ffbffc94, ff3f8aa8, 0, ff3f72c8, 0) + 1050
 ff3d7fe0 _setup   (a, a, ffbfffd6, ffbfffe3, 0, ff3d7c40) + 384
 ff3c2be0 _rt_boot (0, 0, 0, 0, 0, 0) + 88
 00000000 ???????? (0, 0, 0, 0, 0, 0)