Re: Question in regards to debugging programs that use libstdc++-v3...(exceptions & gdb)

["George T. Talbot" <george at moberg dot com>]

Ulrich Drepper wrote:
> 
> "George T. Talbot" <george@moberg.com> writes:
> 
> > (gdb) bt
> > #0  0x4029076e in pthread_handle_sigrestart (sig=32, ctx={gs = 0, __gsh = 0,
> >       fs = 0, __fsh = 0, es = 43, __esh = 0, ds = 43, __dsh = 0,
> >       edi = 3202285892, esi = 1078558464, ebp = 3202285828, esp = 3202285800,
> >       ebx = 3202285892, edx = 3202350760, ecx = 8, eax = 4294967292,
> 
> This normally means the wrong thread is shown.  Do you have a very
> recent gdb with thread support (those using libthread_db)?  Gdb 5
> should work.  Of course you also need a recent system.

I went and checked this, and I don't think so.  Here's the list of threads:

(gdb) info threads
  23 Thread 10916  0x4044e71e in __select () from /lib/libc.so.6
  22 Thread 10915  0x4044e71e in __select () from /lib/libc.so.6
  21 Thread 10894  0x4044e71e in __select () from /lib/libc.so.6
  20 Thread 10893  0x4044e71e in __select () from /lib/libc.so.6
  19 Thread 10884  0x4044e71e in __select () from /lib/libc.so.6
  18 Thread 10882  0x4044e71e in __select () from /lib/libc.so.6
  17 Thread 10876  0x403d1e0e in __sigsuspend (set=0xbdbf01e4)
    at ../sysdeps/unix/sysv/linux/sigsuspend.c:48
  16 Thread 10875  0x403d1e0e in __sigsuspend (set=0xbddf01a4)
    at ../sysdeps/unix/sysv/linux/sigsuspend.c:48
  15 Thread 10874  0x40434371 in __libc_nanosleep () from /lib/libc.so.6
  14 Thread 10873  0x4044e71e in __select () from /lib/libc.so.6
  13 Thread 10872  0x4044e71e in __select () from /lib/libc.so.6
  12 Thread 10871  0x4044e71e in __select () from /lib/libc.so.6
  11 Thread 10870  0x4044e71e in __select () from /lib/libc.so.6
  10 Thread 10869  0x4044e71e in __select () from /lib/libc.so.6
  9 Thread 10868  0x4044e71e in __select () from /lib/libc.so.6
  8 Thread 10867  0x4044e71e in __select () from /lib/libc.so.6
  7 Thread 10866  0x4044e71e in __select () from /lib/libc.so.6
  6 Thread 10865  0x4044e71e in __select () from /lib/libc.so.6
  5 Thread 10864  0x4044e71e in __select () from /lib/libc.so.6
  4 Thread 10863  0x4044e71e in __select () from /lib/libc.so.6
  3 Thread 10862  0x4029076e in pthread_handle_sigrestart (sig=32, ctx={
---Type <return> to continue, or q <return> to quit---
      gs = 0, __gsh = 0, fs = 0, __fsh = 0, es = 43, __esh = 0, ds = 43,
      __dsh = 0, edi = 3212771668, esi = 1078558464, ebp = 3212771604,
      esp = 3212771576, ebx = 3212771668, edx = 3212836520, ecx = 8,
      eax = 4294967292, trapno = 16, err = 0, eip = 1077747214, cs = 35,
      __csh = 0, eflags = 646, esp_at_signal = 3212771576, ss = 43, __ssh = 0,
      fpstate = 0x0, oldmask = 2147483648, cr2 = 0}) at pthread.c:680
  2 Thread 10859 (initial thread)  0x40434371 in __libc_nanosleep ()
   from /lib/libc.so.6
* 1 Thread 10861 (manager thread)  0x4044d410 in __poll (fds=0x809ce00,
    nfds=1, timeout=2000) at ../sysdeps/unix/sysv/linux/poll.c:45
(gdb)

Thread 3 is the thread in question.  I went and did a "bt" on all the
threads.  Threads 23-18 are waiting in select.  Threads 17 & 16 look like
this:

(gdb) bt
#0  0x403d1e0e in __sigsuspend (set=0xbdbf01e4)
    at ../sysdeps/unix/sysv/linux/sigsuspend.c:48
#1  0x40291d14 in __pthread_lock (lock=0x40493934, self=0xbdbffe60)
    at restart.h:32
#2  0x4028f13b in __pthread_mutex_lock (mutex=0x40493924) at mutex.c:84
#3  0x4048368f in find_fde (pc=0x80ad0c0) at ../../gcc/frame.c:626
#4  0x4034660b in throw_helper (eh=0x809dcb8, pc=0x402888c0,
    my_udata=0x401845e8, offset_p=0x401794c7)
   from /usr/local/libstdc++/lib/libstdc++.so.3
(gdb)

Thread 15 is waiting on a condition.  Threads 14-4 are waiting in select. 
Thread 2 is sleeping forever, and thread 1 is waiting in poll.

What I did to force the program to crash was to do ulimit -s 64.  My guess
looking at the backtrace is that the program is trying to restart that thread
because the mutex has been recently unlocked.

The specific problem I'm having with my program is that, when the threads are
just sitting there in a loop like the following, the program will eventually
use up all the stack space and crash.

...
	while (!done)
	{
		try
		{
			pthread_cond_timedwait(queue_not_empty_condition, some_timeout);
			if (timed out)
				throw QueueEmptyException("blah blah blah");

			DequeueAndDoSomethingUseful();
		}
		catch (QueueEmptyException exc)
		{
			if (time_for_this_thread_to_die)
				done	= true;
		}
	}
...

I figured that I'd be clever and reduce the stack space that the program is
using, thus forcing a crash, and then look at the backtrace and see if I did
something stupidly recursive.  Unfortunately, I can't get a decent backtrace.

The weird thing with the bug in my program is that I could swear that I'm not
doing anything except the timeout you see in the pseudo-code above when the
stack is growing.

--
George T. Talbot
<george at moberg dot com>