I have a C++ program that seems to be running into severe memory allocation problems when compiled under g++ 3.4.2 or 3.4.3 , but runs fine when I use the stock Fedora Core 2 g++ 3.3.3 compiler. The actual problem is that the program is 10-20 times slower when compiled using g++ 3.4.2 or 3.4.3 than it is when compiled using g++ 3.3.3. I am submitting this bug report to alert you to this problem. The program's VSZ is around 550Mb, when compiled under either compiler. I see this problem on machines that have either 2Gb or 4Gb RAM, and swapping is not an issue. The program processes a stream of data from gigabyte-sized files (it's actually extracting features from a large number of natural language sentence parses). The program makes two passes over the data; in the first pass it identifies the important features in the data, and in the second pass it writes out which important features occur in each data item. The second pass is a simple loop that reads in each datum, writes out the important features in the datum, and then the temporary data structures involves in the datum are freed; it does not allocate any additional memory. And indeed, the memory used by the program (as measured by ps or top) does not change during the second pass. When the program is compiled using g++ 3.3.3 both passes take approximately the same amount of time to run. However, when the program is compiled using g++ 3.4.2 or 3.4.3 the second pass may take 10-20 or more times longer than the first pass. Moreover, this seems to depend on the amount of data being processed. The problem does not occur when working with smaller data files, for example. When the program is compiled with 3.3.3 both passes take approximately two hours each, but the second pass takes several days to run when compiled with 3.4.3. (The first pass is actually slightly faster with 3.4.3 than under 3.3.3). This also makes it difficult to discover what's happening using gprof, as when compiled the program with profiling turned it is slower still (I killed a profiled version of my program when it hadn't terminated after a week). I'm afraid that this isn't very helpful, and I'd like to do what I can do to help, as I think gcc is a wonderful piece of software and I'd like to contribute if I can. My guess is that the default allocator generally works well, but that the specific pattern of allocation and deallocation that the second pass of my program is performing is somehow very bad for the default allocator. One way to test this would be by changing the default allocator, and see if the behaviour changes. I tried reading the stdlibc++ library documentation on allocators, but I couldn't figure out how to globally change the default allocator. (That used to be possible with a #DEFINE with the old SGI STL, I believe). Is there a way to change the default allocator that my program uses, or to change its configuration parameters in some way so I could try to determine if the allocator really is causing the problem? Anyway, if you have any suggestions for anything else that I might try, I would be pleased to. Thanks, Mark Johnson
the default allocator did change for 3.4.x, from pool_allocator to new_allocator. see: http://gcc.gnu.org/onlinedocs/libstdc++/20_util/allocator.html to change to the 3.3.x allocator, use __pool_alloc. you can configure with --enable-libstdcxx-allocator=pool to make this the default. see: http://gcc.gnu.org/onlinedocs/libstdc++/configopts.html anyway. if you just want to hack in pool for new allocators, locally modify /usr/include/c++/3.4.2/i386-redhat-linux/bits/c++allocator to be src/gcc/libstdc++-v3/config/allocator/pool_allocator_base.h FWIW, it is recommended that you use __mt_alloc for your high-performance allocator needs. -benjamin
I think we can definitely close this one: the various (configure) issues are well known and no fixes are needed anywhere.
Subject: Re: 3.4 default memory allocator much slower than 3.3 allocator when large amounts of data are turned over Yes, although you might want to revert to the 3.3 default allocator, as the 3.4 default allocator is considerably slower for large programs. Best, Mark Johnson pcarlini at suse dot de wrote: >------- Additional Comments From pcarlini at suse dot de 2004-12-31 11:10 ------- >I think we can definitely close this one: the various (configure) issues are well >known and no fixes are needed anywhere. > > >
> Yes, although you might want to revert to the 3.3 default allocator, as > the 3.4 default allocator is considerably slower for large programs. The default 3.4 ""allocator"", actually just forwards to new/delete. Of course, there is a reason why we chose it as default: the new allocators we were working on were not ready in time for 3.4.0; the old one not yet forward ported to the new framework. The 3.4.x ABI is completely frozen, now, and we cannot change the default anymore. For the forthcoming 4.0.0, as indicated by Benjamin, we have a brand new allocator as default, optimized for MT (+ all the old possibilities)