RE: PR java/1373: recursion stress test causes segmentation fault

[Andrew Haley <aph at redhat dot com>]

Boehm, Hans writes:
 > I fully agree with just aborting on stack overflow.  It would be really nice
 > to get that version of the patch in.
 > 
 > I suspect there are more serious issues with throwing StackOverflowError.
 > The problems aren't restricted to the GC, though that's part of it.

Well, right, but the GC issues are the ones I don't already know
about.  So, I'm asking specifically about the GC issues, in particular
my specific question about how the stack is set up, with a write
protected guard area between the stack and the altstack.

 > In fact, thinking about this some more, I'm not sure how this can be
 > safely handled in standard JVMs either.

Exactly.  There aren't any significant differences between a
fully-compiled and a JIT environment from this point of view.  So, if
they can do it, we can do it.

 > In general, a StackOverflowError can occur at arbitrary points,
 > while a global data structure is being updated.  This applies to
 > Java routines, libc routines, and everything in between.  In the
 > case of a libc routine, I think there is no guarantee that even
 > critical locks will be released when we unwind.

Indeed.  I have to assume that other VMs do some sort of probe when
calling libc/kernel routines.

 > For a Java routine, unwinding will usually release the locks, but
 > can leave data structures in an inconsistent state.

Accepted.  It is a stack overflow error, not a stack overflow
exception, so it's not really expected to be restartable.

 > We could conceivably handle the libc and GC issues by always doing
 > a stack probe ahead of time.  But it seems to me that even for Java
 > code, this is basically a bad idea for most of the reasons that
 > Thread.stop() is.  In particular, malicious code could essentially
 > force library code to corrupt its data structures by calling it on
 > a larger and larger stack.

I don't understand how this could happen if we do a stack probe ahead
of time.

 > Thus to guarantee security, library code would have to be written
 > to ensure that it never does anything bad when started in one of
 > these corrupt states.  That strikes me as a hopeless burden.

This is impossible, I think.

 > This is all much less of an issue if you abort on stack overflow.
 > Plus it's a lot easier to implement.
 >
 > (Another issue with StackOverflowError: Is -fnon-call-exceptions
 > sufficient to deal with exceptions on stack accesses, e.g. register
 > spills?)

It depends on when the stack allocation is done.  In general, no.

This will take some time to get right.

Andrew.