Re: GC leaks debugging

[Erik Groeneveld <erik at cq2 dot nl>]

On Sat, Apr 9, 2011 at 3:14 AM, Boehm, Hans <hans.boehm@hp.com> wrote:
> Finally getting around to looking at this thread:

It is very well appreciated.

> It still looks to me like the 32-bit version had serious blacklisting issues, possibly in addition to other problems. ÂIt looks to me like the root sets it's tracing are too large, and they contain too much essentially random data that ends up looking like pointers. ÂThat may not be the only, or even dominant, problem.

Problems seldom come alone...

> But let's continue with the 64-bit version that doesn't share the problem, and should be fine.
>
> What does the GC log look like for the C++ test program you posted? ÂHow much live data is the GC finding?

The log I attached earlier is TestDump012.analyze.tgz (from
gc-analyze) contains the live blocks.  (The TreeMap nodes are not a
problem; they disappear completely later on.)

I only kept the last dump of GC_DUMP_REGULARLY because they became GBs
big.  It is attached.

> That test program is pushing the envelope in a couple of different ways:
>
> 1. It potentially causes lots of fragmentation if some of those objects are not immediately reclaimed. ÂThis garbage collector doesn't like large objects very much. ÂI would expect that even some copying collectors would run into trouble with this, in that copying such large objects is expensive, and they may try to avoid compacting them. ÂOthers would probably do OK.

I am beginning to understand this.

The OpenJDK collector manages it.  The heap stays around 60 Mb.  I

> 2. The heap expansion heuristic in the GC versions < 7 is not very robust in cases like this. ÂIt tries to allocate a fixed fraction of the heap between collections, and grows the heap if that fails. ÂWith large fragmentation, it may never succeed. ÂVersion 7+ fixes this by trying to allocate a fixed fraction of live data, not the overall heap size. ÂThat involves tracking live data, at least approximately. ÂSetting a hard limit on the heap size (GC_MAXIMUM_HEAP_SIZE environment variable) might be a workaround.

I tried different values for GC_MAXIMUM_HEAP_SIZE, but it just runs
until it gives OOM.

I also tried compiling the GC with USE_MUNMAP, as I understand that
that would be an ideal solution if fragmentation is the problem.
However, the program crashes after some time with:

java.lang.ClassCastException: java.util.TreeMap$Node cannot be cast to
java.lang.ref.WeakReference
   at org.apache.lucene.util.CloseableThreadLocal.get(liblucene-core.so)
   at org.apache.lucene.analysis.Analyzer.getPreviousTokenStream(liblucene-core.so)


>> From: Andrew Haley [mailto:aph@redhat.com]
>>
>> > Just two questions:
>> > 1. What is a reasonable number of heap sections? ÂI have 131 here.
>> > 2. What is a reasonable number of free lists? ÂI have 60, which have
>> > 13,000+ entries.
>>
>> Paging Hans Boehm. Âcan you suggest ways to get the system to GC more
>> frequently? ÂWould doing so avoid this scenario?
>>
>> Andrew.
> There is a static, configuration-dependent limit on the number of heap sections. ÂSo long as you're not running out, I wouldn't worry about it.

Ok.

> What kind of free lists are we talking about? ÂIf these are the large block lists printed by GC_dump(), then 13000+ entries sounds really high. ÂIf so, could you post an appropriate excerpt of what's in these free lists? ÂFor small object free lists, that's probably OK.

Yes, I refer to the free lists from GC_dump(), see attachment.

Erik

Attachment: run.lastdump.tar.bz2
Description: BZip2 compressed data