RE: heap fragmentation?

["Boehm, Hans" <hans_boehm at hp dot com>]

The only thing I see here that's clearly suboptimal is the size of static roots.  That basically gets added to the size of live heap data when making heap expansion decisions, since it needs to be scanned for each collection.  We should really be trying to reduce that.  If a lot of it comes from CNI code, you might try calling GC_exclude_static_roots() on pointer-free static data.

Remaining questions: Why did the heap grow to 32 MB?  If you have the complete log, you might ssearch for the heap expansions.  Or you can put a breakpoint in GC_expand_hp_inner().  With a heap size of 20MB, it should insist on allocating something like (20+6)/3 = ~9MB between collections.  With 5 MB in use, that should be fairly easy.

It would be interesting to know whether the heap was fragmented at the point of heap expansion.  The snapshot we're looking at here doesn't look bad to me.  The fact that only 5MB of contiguous free space are available should be OK if your program doesn't allocate huge objects.  You may see a fair number of mostly empty blocks immediately after a GC.  Assuming the allocation pattern of the application doesn't change too much, this shouldn't matter a lot.  Those holes will get filled in before the next GC.

I assume the collection here was triggered explicitly?  Otherwise it shouldn't have been triggered after 25K of allocation.

I don't think gcj ever configures the collector so that the heap can shrink.  Defining USE_MUNMAP should work on some platforms.

It's not clear to me that SMALL_CONFIG will help here.  It might.  Incremental collection generally hurts the memory footprint.

Hans

> -----Original Message-----
> From: Jeff Sturm [mailto:jsturm@one-point.com]
> Sent: Tuesday, June 04, 2002 4:23 PM
> To: java@gcc.gnu.org
> Cc: boehm@acm.org
> Subject: heap fragmentation?
> 
> 
> While testing some applications with 3.1, I've noticed that
> steady-state heap usage is significantly greater than with 
> Sun's VM, by
> roughly a factor of three.  (I would've expected a precise GC 
> to be more
> space efficient than a conservative non-moving collector, but 
> not quite
> this much.)
> 
> Can someone tell me if I understand this output correctly:
> 
> Initiating full world-stop collection 291 after 25312 allocd bytes
> 0 bytes in heap blacklisted for interior pointers
> --> Marking for collection 291 after 25312 allocd bytes + 0 
> wasted bytes
> Collection 290 reclaimed -6736 bytes ---> heapsize = 32661504 bytes
> World-stopped marking took 150 msecs
> Bytes recovered before sweep - f.l. count = -41312
> Immediately reclaimed -41312 bytes in heap of size 32661504 bytes
> 364600 (atomic) + 5346520 (composite) collectable bytes in use
> 
> Total heap size is ~32MB, with 5-6MB in use.  No 
> blacklisting, which is
> good.
> 
> ***Static roots:
> ...
> Total size: 6367157
> 
> ***Heap sections:
> ...
> Total heap size: 32661504
> 
> This concurs with the heap size above.
> 
> ***Free blocks:
> Free list 1 (Total size 2007040):
>     0x101876000 size 8192 not black listed
>     0x101694000 size 8192 not black listed
>     0x101582000 size 8192 not black listed
>     ...
> Free list 2 (Total size 1228800):
> Free list 3 (Total size 737280):
> ...
> Free list 32 (Total size 278528):
> Free list 33 (Total size 360448):
> Free list 60 (Total size 5652480):
> Total of 13369344 bytes on free list
> 
> I see only 5MB or so allocated between collections.  Why does the GC
> maintain more free blocks than needed from one collection to the next?
> Shouldn't MERGE_SIZES help?
> 
> ***Blocks in use:
> (kind(0=ptrfree,1=normal,2=unc.,3=stubborn):size_in_bytes, 
> #_marks_set)
> (4:64,21)(4:64,3)(4:256,25)(1:112,3)(4:96,16)(4:512,16)(4:32,23)
> ...
> blocks = 2268, bytes = 19292160
> 
> These blocks are mostly empty, it seems.  (HBLKSIZE is 8192 on sparc).
> 
> My observations:
> 
> a) A good deal of heap is lying unused.  32MB isn't a large amount,
> however this could increase by at least an order of magnitude 
> if our usage
> patterns change.  Also, significant fragmentation worsens 
> locality, right?
> 
> We're counting on gcj to help reduce our memory overhead.  On the code
> size it certainly does.  But heap size more than offsets that 
> savings, and
> heap can't be shared with other processes.
> 
> b) Building with SMALL_CONFIG should help, as does incremental
> collection.  But those two cannot be used together.
> 
> c) Presently I'm building 64-bit executables, which no doubt 
> contributes
> to heap size.  The VM I'm comparing to is a 32-bit VM, which 
> isn't exactly
> a fair comparison.  However, byte/char arrays are responsible 
> for a good
> part of the heap, which you don't see above because I don't have hash
> synchronization enabled.
> 
> d) Hash synchronization will help by trimming 8 bytes from 
> each object.
> 
> Any other strategies to avoid fragmentation?  I couldn't find 
> much in the
> documentation available.
> 
> Jeff
>