Re: [RFC] 12658_thread.cc randomly fails (on MP machines)

[Paolo Carlini <pcarlini at suse dot de>]

Benjamin Kosnik wrote:

> > I'm trying to understand more about this issue: as you may have
> > noticed already, on multiprocessor machines the testcase randomly
> > fails. This happens both for mainline and 3_4-branch.
> >    
> I have seen some oddness. I didn't know if this was just something that
> should be fixed in the existing testcase, via number of iterations or
> whatever.
>  
Indeed the testcase, especially when run on MP machines, doesn't seem 
very specific...

> I cannot reproduce the fail here, or on machines I have access to.
>  
Which architectures? You need a multiprocessor machine, but otherwise is 
pretty
easy to reproduce! Have a look to testresults, there are many reported 
fails on x86_64
and ia64, for instance.

> > Backtraces on a 4-way x86_64 machine show that the problem seems
> > *not* due to locale::locale() and locale::global(const locale&),
> > which appear indeed fixed: the testcase stresses the MT-safety of
> > the locale implementation *well* beyond what originally reported
> > in libstdc++/12658.
> >    
> Well, what does the backtrace say?
>  
Of course, as expected for this kind of problem, there is a great deal 
of randomness
and variability. This is a rather common pattern, on 4-way x86_64, for
12658_thread as-is, mainline:

(gdb) run
Starting program: /suse/pcarlini/Work/12658/a.out
[Thread debugging using libthread_db enabled]
[New Thread 182903063872 (LWP 19876)]
[New Thread 1075837280 (LWP 19879)]
[New Thread 1077934432 (LWP 19880)]
[New Thread 1080031584 (LWP 19881)]
[New Thread 1082128736 (zombie)]
[New Thread 1084225888 (zombie)]
[New Thread 1086323040 (zombie)]
[New Thread 1088420192 (zombie)]
[New Thread 1090517344 (zombie)]
[New Thread 1092614496 (LWP 19887)]
[New Thread 1094711648 (LWP 19888)]
[New Thread 1096808800 (LWP 19889)]
[New Thread 1098905952 (LWP 19890)]
[New Thread 1101003104 (LWP 19891)]
[New Thread 1103100256 (LWP 19893)]
[New Thread 1105197408 (LWP 19894)]
[New Thread 1107294560 (LWP 19895)]
[New Thread 1109391712 (LWP 19898)]
[New Thread 1111488864 (LWP 19899)]
[New Thread 1113586016 (LWP 19900)]
[New Thread 1115683168 (LWP 19901)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1107294560 (LWP 19895)]
0x0000002a95bf9474 in _nl_remove_locale () from /lib64/tls/libc.so.6
(gdb) backtrace
#0  0x0000002a95bf9474 in _nl_remove_locale () from /lib64/tls/libc.so.6
#1  0x0000002a95bfacf5 in freelocale () from /lib64/tls/libc.so.6
#2  0x0000002a95816801 in ~messages (this=Variable "this" is not 
available.) at messages_members.h:69
#3  0x0000002a957d4516 in std::locale::facet::_M_remove_reference 
(this=Variable "this" is not available.) at locale_classes.h:411
#4  0x0000002a957d4769 in ~_Impl (this=Variable "this" is not available.)
   at /usr/src/paolo/gcc-head/gcc/libstdc++-v3/src/locale.cc:213
#5  0x0000002a957d51bd in ~locale (this=Variable "this" is not 
available.) at locale_classes.h:516
#6  0x0000000000401303 in thread_main () at 12658_thread.cc:50
#7  0x0000002a956738e2 in start_thread () from /lib64/tls/libpthread.so.0
#8  0x0000002a95c8b0a3 in thread_start () from /lib64/tls/libc.so.6
#9  0x0000000000000000 in ?? ()
#10 0x0000000000000000 in ?? ()
#11 0x0000000000000000 in ?? ()
#12 0x0000000000000000 in ?? ()
#13 0x0000000000000000 in ?? ()
#14 0x0000000000000000 in ?? ()
#15 0x0000000000000000 in ?? ()
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000000 in ?? ()
#18 0x0000000000000000 in ?? ()
#19 0x0000002a95dd84c0 in _nl_C_locobj () from /lib64/tls/libc.so.6
#20 0x0000000041fffdc0 in ?? ()
...

Before you ask ;), I have already checked that the problem has nothing 
to do specifically with the messages facet: if I hack it a bit, then 
happens at destruction time for another facet...

Reducing max_thread_count to 2 makes the Segmentation fault less 
frequent (perhaps 1 every 30 runs?).

An example for 3_4-branch, again 12658_thread as-is:

(gdb) run
Starting program: /suse/pcarlini/Work/12658/a.out
[Thread debugging using libthread_db enabled]
[New Thread 182903076160 (LWP 21178)]
[New Thread 1075837280 (LWP 21181)]
[New Thread 1077934432 (LWP 21182)]
[New Thread 1080031584 (LWP 21183)]
[New Thread 1082128736 (zombie)]
[New Thread 1084225888 (LWP 21185)]
[New Thread 1086323040 (LWP 21186)]
[New Thread 1088420192 (LWP 21187)]
[New Thread 1090517344 (LWP 21188)]
[New Thread 1092614496 (LWP 21189)]
[New Thread 1094711648 (LWP 21190)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 1084225888 (LWP 21185)]
0x0000002a95bfc784 in _nl_find_locale () from /lib64/tls/libc.so.6
(gdb) backtrace
#0  0x0000002a95bfc784 in _nl_find_locale () from /lib64/tls/libc.so.6
#1  0x0000002a95bfd7e0 in newlocale () from /lib64/tls/libc.so.6
#2  0x0000002a95830bbe in std::locale::facet::_S_create_c_locale 
(__cloc=@0x409ff630, __s=0x0,
   __old=0xffffffffffffffff) at c++locale.cc:99
#3  0x0000002a957d94a6 in _Impl (this=0x2a96005cc0, __s=0x40155c "en_US",
   __refs=18446744073709551615)
   at /usr/src/paolo/gcc-3_4-branch/gcc/libstdc++-v3/src/localename.cc:189
#4  0x0000002a957da6ed in locale (this=0x409ff750, __s=0x40155c "en_US")
   at /usr/src/paolo/gcc-3_4-branch/gcc/libstdc++-v3/src/localename.cc:46
#5  0x000000000040102d in thread_main () at 12658_thread.cc:40
#6  0x0000002a956738e2 in start_thread () from /lib64/tls/libpthread.so.0
#7  0x0000002a95c8e0a3 in thread_start () from /lib64/tls/libc.so.6
#8  0x0000000000000000 in ?? ()
#9  0x0000000000000000 in ?? ()
#10 0x0000000000000000 in ?? ()
#11 0x0000000000000000 in ?? ()
#12 0x0000000000000000 in ?? ()
#13 0x0000000000000000 in ?? ()
#14 0x0000000000000000 in ?? ()
#15 0x0000000000000000 in ?? ()
#16 0x0000000000000000 in ?? ()
#17 0x0000000000000000 in ?? ()
...

> > If, less ambitiously (at least for now!) we want only MT-safety
> > in dealing with the global locale, then probably we are already
> > quite close but we badly need a different, much simpler, testcase
> > than 12658_thread.
> >    
> This is probably the place to start.
>  
Agreed. Thanks.

Paolo.