Re: Heap fragmentation (Was: Debugging "Leaks" With Boehm-GC)

[Hans Boehm <Hans dot Boehm at hp dot com>]

On Sat, 14 Jan 2006, David Daney wrote:

> [Hans wrote:]
> >  Really fragmentation per
> > se can only occur if either:
> >
> > 1) The application drastically changes the object size mix it needs for
> > different phases, and some other things go wrong.  And even then, things
> > shouldn't get too bad.  Or
> >
>
> For my this was the problem.
>
> For certain definitions of 'drastically' and for heaps of certain size
> limits using the default divisor: "shouldn't get too bad" is equivilent
> to "out of memory".
>
> Now I don't know if you would call it classical memory fragmentation,
> but what ends up happening is that the memory pools for small sized
> objects grow to such a point that allocation of objects of a newly seen
> size fails.  There is plenty of space for objects for which there is a
> large pool.
I think for this to fail as described, you need not only need to allocate
lots of small objects as you describe, but then also drop most of them,
but leave at least one live in each heap chunk/block.  Otherwise the entire
block would be reclaimed, and it should be usable for large objects again.

That's certainly possible.  It's just not something I've seen a lot of.
But it does explain why the problem gets much better with a larger
GC_free_space_divisor, which was a bit of a mystery to me before.

>
> I forget the actual algorythm the GC uses, but IIRC it only considers
> overall memory usage when deciding when to run a GC cycle.  For limited
> memory situations, it might be better to also consider the pool size in
> relation to the total available memory and not led the pools take too
> much memory.  Perhaps if a certain pool is larger than 5-10% of the
> total available memory always run GC instead of expanding the pool.  In
> effect a dynamic divisor based on pool size.
I expect it would be hard to make that robust, since some applications
are bound to allocate primarily one object size.  But it might be an
interesting option.
>
> I should also note that all my research into these problems was with
> libgcj-3.3.1.  Since then we have always used a larger divisor and never
> had any more problems.
>
> David Daney
>
>
>