"Out of Memory" crash

["Scott Gilbertson" <scottg at mantatest dot com>]

My AWT/xlib application (using gcc 3.2.1 with libjava patches, on
linux/i386, statically linked) crashes pretty easily under load (i.e., when
a lot of paint events are occurring).  Looking in /var/log/kernel, I see
messages like:
    ... kernel: Out of Memory: Killed process 123 ...
If you look at MemFree (in /proc/meminfo) while the application is painting
like mad, you see the number declining, but if you stop the action for a few
seconds, MemFree pops right back up.  This tells me that the memory use
which is causing the crash is collectable, but the GC is only collecting it
if you stop or reduce the action for a while (either that, or the objects
become collectable only when there's a break, but that seems unlikely to
me).

Shouldn't the Boehm GC collect a bunch of these objects whenever it runs out
of memory while trying to allocate something, rather than allowing the
application to be killed?

I have read some earlier postings regarding GC_max_retries.  I see
GC_max_retries=0 in alloc.c.  Does this have anything to do with the
problem?  If so, why is it zero by default?  Where is a good place to put
"GC_max_retries=n", and what would be a good value for n?

I tried changing GC_max_retries in alloc.c, with no improvement.  I don't
think I'm building it right, though -- I noticed that
/usr/lib/libgcjgc.so.1.0.1 doesn't get updated when I do "make install" from
the gcc, libjava or boehm-gc build directory.  Is there a trick to
installing modified boehm-gc code?

I tried doing a System.gc every 10 user input events (about 2 per second at
full tilt).  This change makes it harder, but not impossible, to crash the
app by flooding it with events that repaint things.  Once MemFree starts to
drop, even System.gc doesn't seem to stop it.  Of course my app is basically
unusable with that many full garbage collections going on.

Also, is there a problem with Runtime.totalMemory / _Jv_GCTotalMemory?  It
shows numbers in the 2-4 megabyte range, but the crash occurs when the
application has decreased meminfo/MemFree by something like 60-70 megabytes.
If _Jv_GCTotalMemory is OK, maybe this discrepancy indicates that there's a
lot of stuff being allocated and freed in native code, bypassing the GC.

Perhaps it would help to limit the memory available to my application, but I
don't know how to do that.  I tried "ulimit -l" and "ulimit -m", but they
don't seem to have any effect on my target system.  Is there a way to set
the maximum heap size for a gcj-compiled executable?