This is the mail archive of the libstdc++@gcc.gnu.org mailing list for the libstdc++ project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: stl / pthreads leak, patch


In article <1029204073.1359.22.camel@penguin1>,
Kenneth Murray<desertcoder@cox.net> writes:

Hi Ken, based on your last communications and thinking about it some
more, I see that there are a few points that I should have attached to
my analysis of your submitted test cases.  Here are those points: I
have looked at your test cases on i386-*-freebsd* (what I have
available) not i386-*-linux* (where you have seen the problem).  I
looked at them with gcc mainline and gcc 3.1 (again, because that is
what I have built locally, 3.1.1 is somewhere in between those).  I
did make the test cases deterministic, steady-state before running
them with systemic memory leak tools I have available.  I assure you
that I would have spotted a systemic memory leak, which is the only
kind we will consider to be a libstdc++-v3 bug, had one existed in the
code you sent me.

Now, it is possible that our libc and pthread libraries behave quite
differently...but I have never thought that this was the source of the
on-and-off "memory leak" reports we get.  To further test my theories,
I will post a version of your second test case that has been converted
to deterministic, halting.  Would someone on Linux care to report
findings of how it performs under ``time'' and ``valgrind''?

#define _GNU_SOURCE
#define _REENTRANT
#define _THREAD_SAFE
#include <pthread.h>
#include <string>
#include <unistd.h>

using namespace std;

int strMultiplier = 100; 

void *processClient(void *clientArgs) 
{       
        string s1, s2, s3, s4, s5, s6;
        s1.assign("blah123");
        s2.assign("blah123blah123");
        s3.assign("blah123blah123blah123");
        s4.assign("blah123blah123blah123blah123");
        
        int r = 5  * strMultiplier;
        for ( int i = 0; i <= r; i++  ) {
                s5.assign(s1 + s2);
                s1.assign(s5);
        }               
        s6.assign(s3 + s2 + s1);
                
        return NULL; 
}

int main(int argc, char **argv)
{
  pthread_t clientThread[40];

  for (int i = 0; i < 40; i++)
    if ((pthread_create(&clientThread[i], NULL, processClient, NULL)) != 0)
      return 1;
  for (int i = 0; i < 40; i++)
    pthread_join (clientThread[i], NULL);

  return 0;
}

Here are results for gcc 3.1 (gcc mainline similar) with
``strMultiplier = 100'' for a PII-350MHz i386-*-freebsd* machine (of
course, g++ reports ``Thread model: posix''):

/usr/bin/time -l a.out
        0.23 real         0.22 user         0.00 sys
      1236  maximum resident set size
         8  average shared memory size
       313  average unshared data size
       132  average unshared stack size
       165  page reclaims
         0  page faults
         0  swaps
         0  block input operations
         0  block output operations
         0  messages sent
         0  messages received
        22  signals received
         0  voluntary context switches
         4  involuntary context switches

My predication is that valgrind will report no "lost pointer" leaks.
On Ken's hardware/OS, my prediction is that this test case requires
more than 10 seconds when run under valgrind.  However, there may be a
subtle difference between this case and the one he posted such that
this one runs faster.

Also, I see this test case isn't fair since on FreeBSD, the
newly-created threads usually run to completion before main() starts
the next thread (I know this from the low signal count) thus there was
no contention.  Here is a non-trivial result for ``strMultiplier = 1000'':

/usr/bin/time -l a.out
       36.28 real        35.49 user         0.42 sys
      7376  maximum resident set size
         8  average shared memory size
      3827  average unshared data size
       128  average unshared stack size
      8589  page reclaims
         0  page faults
         0  swaps
         0  block input operations
         0  block output operations
         0  messages sent
         0  messages received
      3603  signals received
         0  voluntary context switches
       576  involuntary context switches

> I am well aware of the true definition of a "memory leak".

I fear that you have taken insult where none was intended.  In any
event, I do not believe that there is one true definition of a memory
leak.  FYI, the explicit analysis details of my post which included
these definitions were to correlate what valgrind reports with
"white-box" source analysis not to insult anyone's knowledge of
definitions of "memory leak".  It is clear to me that many people use
the words "memory leak" without sharing context of what they mean.

> The problem is either that 1) the "cache" grows larger and larger
> without a limit, or 2) the stl looses track of something and does
> not perform enough frees for each alloc. The reporting I get from
> Valgring seems to support #2.

I disagree with your analysis.  Your analysis is wrong IMHO since you
have looked at cases where threads are killed in a non-deterministic
manner with non-deterministic state.  Anytime you kill a threaded
process with non-joined, detached threads (such as when you fall off
the end of main() without joining all spawned threads) it is a
non-deterministic case and it could appear to have many such "memory
leaks" as valgrind reports.  I believe that I covered this as a source
of your finding in my theory in the first analysis and based on past
experience.  BTW, I believe (but I will not look up a cite) that
someone else had this exact problem last year and eventually came
around to my thinking once they looked at deterministic test cases...

> In any case, it does not look like the stl is thread-safe for
> applications that 1) are suppose to run "forever", 2) make heavy use
> of the stl, and 3) make heavy use of pthreads.

If this statement were true, someone should be able to point to the
exact lines of the library with a bug.  Or, at least, provide a test
case that exposes the bug.  Unless there is a bug, the internal cache
for STL and small strings will stabilize at the maximum memory profile
ever used until that point in the run.

> I know several others have reported this bug. Maybe someone, someday
> will fix it. Until then, I have no other choice but to abandon the stl
> when using pthreads. 

What bug?  No person that has reported this "bug" has ever been able
to provide a test case that shows it.  Your reports have not shown any
issue with the library code proper (at least not anything which is
platform independent, and I'm not pointing a finger at glibc with this
statement or whatever since I don't think enough data exists).  You
are more than welcome to reconfigure it to remove the caching
behavior, if you consider that a bug or misfeature (however, that same
misfeature was in libstdc++-v2, although use of the STL cache
architecture by small strings was not in place).  Unless one puts
non-trivial effort into the cache architecture, it will not be able to
free stuff back to the OS.

BTW, I do not doubt that many people, especially on i386-*-linux*,
including you Ken, have seen some problem which they attribute to the
way libstdc++-v3 is tuned or built on their system but not once have
we received information enough to actually change it.  With the right
study, we might be able to figure out the disconnect and actually fix
it (either with better documentation or an actual library change).
Nothing would please me more.

Regards,
Loren


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]