Program terminated with signal 11, Segmentation fault in do_lookup_x

[liangfan]

Hi,
I got some core dumps of my code due to signal 11, Segmentation fault in do_lookup_x.
 
The running environment is:
2.6.24.7-9.smp.gcc3.4.x86_64
glibc=2.3.6-10-1
 
Below is the detailed back strace:
----------------------------------------
(gdb) bt full
#0  do_lookup_x (undef_name=0x2af785f9cbee "__errno_location", hash=202084270, ref=0x2af785f9ca90, result=0x4b013770, 
scope=0x15, i=10, 
    version=0x2af787c4ac80, flags=1, skip=0x0, type_class=1) at do-lookup.h:78
 symtab = (const Elf64_Sym *) 0xb00a
 strtab = 0xb000 
 verstab = (const Elf64_Half *) 0x2af786f8a6f6
 symidx = 1066
sym = (const Elf64_Sym *) 0x113fa
 num_versions = 0
 versioned_sym = (const Elf64_Sym *) 0x0
 list = (struct link_map **) 0x2af787c4a830
 n = 21
 map = (struct link_map *) 0x2af78666f980
 __PRETTY_FUNCTION__ = "do_lookup_x"
#1  0x00002af784d70441 in _dl_lookup_symbol_x (undef_name=0x2af785f9cbee "__errno_location", undef_map=0x2af785949960, 
ref=0x4b0137d8, 
    symbol_scope=0x2af785949ca8, version=0x2af787c4ac80, type_class=1, flags=1, skip_map=0x0) at dl-lookup.c:232
 res = 22
 start = 3198
 hash = 202084270
 current_value = {s = 0x0, m = 0x0}
 scope = (struct r_scope_elem **) 0x2af785949ca8
 __PRETTY_FUNCTION__ = "_dl_lookup_symbol_x"
 i = 0
 protected = -2022338828
#2  0x00002af784d73164 in fixup (l=0x2af785949960, reloc_offset=45066) at dl-runtime.c:98
 version = (const struct r_found_version *) 0x16
 strtab = 0x15 
 reloc = (const Elf64_Rela * const) 0xb00a
 sym = (const Elf64_Sym *) 0x2af785f9ca90
 rel_addr = (void * const) 0x2af7860a2e58
 result = 0x2af787c4a830
 value = 3198
 __PRETTY_FUNCTION__ = "fixup"
#3  0x00002af784d73012 in _dl_runtime_resolve () at dl-runtime.c:62
 stack_pointer = 0x4b0137f0 ""
#4  0x00002af785f9f3dd in log_message () from /usr/lib64/libcommonlog.so.1
No symbol table info available.
#5  0x00002af785fa0e12 in logger_log () from /usr/lib64/libcommonlog.so.1
No symbol table info available.
#6  0x00002af78515c94d in Net::SendThread (arg=0x2af785f9ce43) at tracer.hxx:88
 conn = (Conn *) 0x2aaaab6113d0
 __FUNCTION__ = "SendThread"
#7  0x0000000000000fff in ?? ()
No symbol table info available.
#8  0x0000000000001000 in ?? ()
No symbol table info available.
#9  0x00002af78792335a in start_thread () from /lib64/tls/libpthread.so.0
No symbol table info available.
#10 0x00002af7877ae473 in clone () from /lib64/tls/libc.so.6
 fstab_state = {fs_fp = 0x0, fs_buffer = 0x0, fs_mntres = {mnt_fsname = 0x0, mnt_dir = 0x0, mnt_type = 0x0, mnt_opts = 0x0, 
    mnt_freq = 0, mnt_passno = 0}, fs_ret = {fs_spec = 0x0, fs_file = 0x0, fs_vfstype = 0x0, fs_mntops = 0x0, fs_type = 0x0, 
fs_freq = 0, 
    fs_passno = 0}}
 __elf_set___libc_subfreeres_element_fstab_free__ = (const void *) 0x2af7877df690
 
(gdb) f 0
(gdb)info registers
info register
rax            0xc7e 3198
rbx            0x42a 1066
rcx            0x2af787c4a830 47242623100976
rdx            0x16 22
rsi            0xb00a 45066
rdi            0x2af785f9ce43 47242593029699
rbp            0x2af78666f980 0x2af78666f980
rsp            0x4b0135e0 0x4b0135e0
r8             0x15 21
r9             0x0 0
r10            0x2af785949960 47242586397024
r11            0x20 32
r12            0x113fa 70650
r13            0x42a 1066
r14            0xa 10
r15            0x22 34
rip            0x2af784d700aa 0x2af784d700aa 
eflags         0x10202 66050
cs             0x33 51
ss             0x2b 43
ds             0x0 0
es             0x0 0
fs             0x63 99
gs             0x0 0
---------------------------------------
According to the bt, it cores at do-lookup.h 78 
75   sym = &symtab[symidx];
76
77   assert (ELF_RTYPE_CLASS_PLT == 1);
78  if ((sym->st_value == 0 /* No value.  */
And undef_name is "__errno_location" seems it try to find error_no.
However, I can not figure out what cause this.
I tried to use Valgrind and fortify, but can not find any error or info may relate to this.
Besides, I got 8 core dumps on 8 nodes at almost the same time. After that ,the service restarted, and run about 1 week till 
now and the core dump does not happen again.
I am just look at the dynamic loading of the lib. 
Any ideas or suggestion on this will be really appreciated.
More info will be added if needed.
 
Thanks!
Fan
_________________________________________________________________
ÕÅÈýÍÚµ½ÁËÔª±¦,СÃÀÓÖµô½øÏÝÚåÁË,¿ìÀ´MClubÓëºÃÓÑÆëÀÖÀÖ£¡Á¢¿Ì·ÃÎÊ£¡
http://club.msn.cn/?from=3