[Bug libstdc++/18970] New: 3.4 default memory allocator much slower than 3.3 allocator when large amounts of data are turned over

["mj1 at cog dot brown dot edu" <gcc-bugzilla at gcc dot gnu dot org>]

I have a C++ program that seems to be running into severe memory allocation
problems when compiled under g++ 3.4.2 or 3.4.3 , but runs fine when I use the
stock Fedora Core 2 g++ 3.3.3 compiler.  The actual problem is that the program
is 10-20 times slower when compiled using g++ 3.4.2 or 3.4.3 than it is when
compiled using g++ 3.3.3.  I am submitting this bug report to alert you to this
problem.

The program's VSZ is around 550Mb, when compiled under either compiler.  I see
this problem on machines that have either 2Gb or 4Gb RAM, and swapping is not an
issue.

The program processes a stream of data from gigabyte-sized files (it's actually
extracting features from a large number of natural language sentence parses).
The program makes two passes over the data; in the first pass it identifies the
important features in the data, and in the second pass it writes out which
important features occur in each data item.  The second pass is a simple loop
that reads in each datum, writes out the important features in the datum, and
then the temporary data structures involves in the datum are freed; it does not
allocate any additional memory.  And indeed, the memory used by the program (as
measured by ps or top) does not change during the second pass.

When the program is compiled using g++ 3.3.3 both passes take approximately the
same amount of time to run.  However, when the program is compiled using g++
3.4.2 or 3.4.3 the second pass may take 10-20 or more times longer than the
first pass.  Moreover, this seems to depend on the amount of data being
processed.  The problem does not occur when working with smaller data files, for
example.  When the program is compiled with 3.3.3 both passes take approximately
two hours each, but the second pass takes several days to run when compiled with
3.4.3.  (The first pass is actually slightly faster with 3.4.3 than under
3.3.3).    This also makes it difficult to discover what's happening using
gprof, as when compiled the program with profiling turned it is slower still (I
killed a profiled version of my program when it hadn't terminated after a week).

I'm afraid that this isn't very helpful, and I'd like to do what I can do to
help, as I think gcc is a wonderful piece of software and I'd like to contribute
if I can.

My guess is that the default allocator generally works well, but that the
specific pattern of allocation and deallocation that the second pass of my
program is performing is somehow very bad for the default allocator.

One way to test this would be by changing the default allocator, and see if the
behaviour changes.  I tried reading the stdlibc++ library documentation on
allocators, but I couldn't figure out how to globally change the default
allocator.  (That used to be possible with a #DEFINE with the old SGI STL, I
believe).  Is there a way to change the default allocator that my program uses,
or to change its configuration parameters in some way so I could try to
determine if the allocator really is causing the problem?

Anyway, if you have any suggestions for anything else that I might try, I would
be pleased to.

Thanks,

Mark Johnson

-- 
           Summary: 3.4 default memory allocator much slower than 3.3
                    allocator when large amounts of data are turned over
           Product: gcc
           Version: 3.4.3
            Status: UNCONFIRMED
          Severity: minor
          Priority: P2
         Component: libstdc++
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: mj1 at cog dot brown dot edu
                CC: gcc-bugs at gcc dot gnu dot org,mj1 at cog dot brown dot
                    edu
 GCC build triplet: gcc-3.4.3  --with-arch=pentium4 --enable-
                    languages=c,c++,f77  po
  GCC host triplet: Linux 2.6.9-1.6_FC2smp #1 SMP Dual Pentium 4 Xeon


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=18970