Re: Regression test for thread safety?

[Dan Kegel <dkegel at ixiacom dot com>]

Andreas Jaeger <aj at suse dot de> wrote:
> dank@kegel.com writes:
> > For instance, I downloaded 
> >  ... gcc/releases/gcc-3.1/gcc-testsuite-3.1.tar.bz2
> > unpacked it, and did
> >    find . -print | xargs grep pthread_create
> > and saw *zero* matches. ...
> It might be that gcc-testsuite-3.1.tar does not contain the libstdc++
> testsuite.  A CVS checkout shows the following:
>
> gromit:/cvs/gcc-3_1-branch/libstdc++-v3:[0]$ grep -r pthread_create .
> ./testsuite/thread/pthread1.cc:   pthread_create (&prod[i], NULL, produce, static_cast<void*> (tq[i]));
Thanks!  The tests are indeed in ... gcc/releases/gcc-3.1/gcc-g++-3.1.tar.bz2

I did some experiments to measure how effective these tests are
at detecting thread safety problems.  First, I tested whether
compiling with -D_NOTHREADS disabled locking.  As expected,
 g++-3.0 -E t.C | grep -i __lock_instance
showed hits,  but adding -D_NOTHREADS showed no hits.

Then I compiled the tests pthread[1-6].cc with -D_NOTHREADS, and
ran them ten times on two different machines: a 1.4GHz dual CPU box (pentum 4)
and a 1.4GHz UP box (pentium 3), both running "Debian Woody testing".

I'd like to emphasize that all these tests should fail, since I
turned off locking!  The question is, how well do the tests detect
the missing thread safety.

The first thing I learned was that my SMP pentium 4 box was a
whole lot faster at running these tests than my UP pentium 3 box :-)

The second thing I learned was that on UP, test pthread4.cc
runs very, very slowly; it would have taken all day to run,
so I was unable to use it on my UP machine.

The third thing I learned was that on UP, test pthread5.cc
had a failure mode where several of its threads would
finish, leaving one or more others running forever.
For now, I detected that manually and killed the test by hand,
counting that as a failure.  A monitor program might be able to
do that automatically.


Results:

# of crashes in 10 runs:

pgm#    SMP    UP
1 
10 
2
2 
5	0
3 
4	0
4 
1 
0 (gave up on up)
5 
10 
4 (with manual timeout on up twice)
6 
10	3

Hopefully this data will be helpful for others planning to
use these regression tests to go looking for thread support
problems on homebrew gcc installations.  If you have limited
time, and need to find a thread support problem on a uniprocessor,
the data suggests running pthread5.cc, pthread6.cc, and pthread1.cc
will be more effective than running pthread[234].cc.

Now I'll try the same thing on the single-processor embedded system
that is having an STL problem... maybe I'll find I built gcc incorrectly.

Cheers,
Dan