Looking for ideas to fix X server crash running GCJ-compiled program

["Scott Gilbertson" <scottg at mantatest dot com>]

Recent tests of my application cause the X server to crash.  I don't know if
it's a problem with the server, the peers, awt, the garbage collector or the
optimizer (or something else).  I'm hoping this problem sounds really
familiar to somebody, because I'm running out of ideas.

It all started when I tried to get my app running without any "System.gc"
calls, because they cause disgusting pauses.  That meant I had to get the
garbage collector to stop using up all the RAM in my system, so I made a
native function that does:
  _Jv_GCSetMaximumHeapSize ((size_t)bytes);
  _Jv_GCSetInitialHeapSize ((size_t)bytes);
  GC_max_retries = 999999999L;
where 'bytes' is typically 20,000,000.  Now when I do a "load test", which
involves simulating about 200 user input events per second with a function
generator hooked up to a rotary knob input, the program doesn't run out of
memory, so it runs much longer than an earlier attempt (several minutes!).
The heap use periodically gets near the 20 meg limit then backs off to 13-15
megs and climbs again.  It climbs quickly, because there's garbage generated
every time I put text on the screen -- I'll be looking at ways to reduce
garbage generation (mainly implementing unimplemented methods in the xlib
peers) once I get it to stop crashing.  I mention all this because I wonder
if the collection or finalization of some object is somehow involved in the
X server crash.  I didn't seem to have the crash before I took out the
System.gc calls.

I'm running linux kernel 2.4.19 on i386.  My gcc is a 3.3 snapshot from a
couple of weeks ago, with all my awt/xlib patches (even the one Tom doesn't
like, to permit lightweight Containers).  I'm using the xlib peers.

Now the X server crashes (taking my application with it, of course) after a
few minutes of high load.  According to a gdb backtrace, the problem is a
SEGV in CopyGC.  I'm using Xfbdev (tiny-X with frame-buffer driver).  I
built and tested the new version 4.3 XFree86 (previous tests were on 4.2.0),
with the same result.  I tried gdb'ing each of the 8 processes in my
application, but I don't get any consistent point of failure.  The program
gets a SIGHUP, which I've figured out means the crash is in the X server.
I'm not sure how to debug the problem, and I'm hoping it's obvious to
somebody.

I looked in the peer code and the only place XCopyGC (the client-side
function which triggers CopyGC) is called is in gnu/gcj/xlib/natGC.cc.

Here are my theories:
1. CopyGC isn't checking its arguments enough, and we're generating a bad
argument.
2. We are prematurely collecting an object from which CopyGC is trying to
get data.
3. We are calling XFreeGC and subsequently using that GC (i.e., cloning a GC
that's been disposed), and the server doesn't catch the error.
4. The optimizer is re-arranging code, causing #2 or #3.
5. We are using too many X server resources before freeing some

I guess #1 would be an XFree86 bug, and doesn't seem all that likely given
the maturity level of XFree86.  On the other hand, I'm using a tiny-X build,
and maybe that eliminates some of the error checking.

I don't know enough to decide whether #2 is a real possibility.  For
example, it's not clear to me whether the X server digs directly into the
client's memory when both are running on the same machine, and hence would
get upset if you'd reclaimed the object right after calling XCopyGC, but
before the server has processed the request.  That's what's implied at
http://tronche.com/gui/x/xlib/my-own/gcontext.html : "GCs however, are
references to opaque structures stored on the client side for efficiency
reasons."

For #3, I inspected the code, and couldn't spot any place where we use a GC
after disposing it.

For #4, I'm thinking of the cases where we have something like:
  SomeClass lThing = thing;
  thing = null;
  if (lThing != null)
    lThing.dispose ();
You'd obviously want the nullification to happen before the dispose.  An
over-enthusiastic optimizer could theoretically reverse that.  It doesn't
seem very likely, though.

For #5, I inspected the code.  The only thing I noticed is that
java.awt.Container.visitChild() creates (i.e. clones) a Graphics and doesn't
dispose it.  I added gfx2.dispose() in a finally block, but it didn't help.
Here's the modified code:
  private void visitChild(Graphics gfx, GfxVisitor visitor,Component comp)
  {
    Rectangle bounds = comp.getBounds();
    Rectangle clip = gfx.getClipBounds().intersection(bounds);

    if (clip.isEmpty()) return;

    Graphics gfx2 = gfx.create();
    try
    {
      gfx2.setClip(clip.x, clip.y, clip.width, clip.height);
      gfx2.translate(bounds.x, bounds.y);

      visitor.visit(comp, gfx2);
    }
    finally
    {
      gfx2.dispose ();    // This is the thing I added
    }
  }


Any suggestions how I should proceed on debugging this thing, or theories
that I've missed?
Thanks.