RE: PR java/1373: recursion stress test causes segmentation fault

["Boehm, Hans" <hans dot boehm at hp dot com>]

My main concern here is that we're potentially investing lots of effort in complying with what turns out to be a security hole in the Java spec.  If that's the case, I think the effort is better invested in getting the spec fixed.

As far as I can tell, there is no security check that guards against malicious code catching an error such as StackOverflowError().  Thus malicious code can force trusted code (executing some harmless operation for which the security check passed) to abort in the middle of a critical section and then regain control, possibly retrying as necessary until it sees the corrupted state it wants.  This is probably nontrivial to exploit, but I'd be surprised if it were impossible to exploit.  And it appears to make it impractical to prove any interesting security properties.

As Andrew points out, we can do a stack probe at the beginning of each function to ensure that StackOverflowError is only thrown there.  (Since StackOverflowError is supposed to be precise, like other Java exceptions, I suspect something like this is basically unavoidable.)  But that doesn't help much.  The trusted code can still potentially abort at every call site.  And global data structures aren't consistent at every call site.  And I can't find any documentation about where StackOverflowError is allowed to be raised, so the author of trusted library code has no idea what the code has to guard against anyway.

Unless I'm misunderstanding something, this looks broken to me.  I no longer see a good reason to go there.

Hans



> -----Original Message-----
> From: java-patches-owner@gcc.gnu.org
> [mailto:java-patches-owner@gcc.gnu.org]On Behalf Of Andrew Haley
> Sent: Thursday, October 21, 2004 7:44 AM
> To: Anthony Green
> Cc: Boehm, Hans; java-patches@gcc.gnu.org
> Subject: RE: PR java/1373: recursion stress test causes segmentation
> fault
> 
> 
> Anthony Green writes:
>  > On Thu, 2004-10-21 at 01:35, Andrew Haley wrote:
>  > >  > In fact, thinking about this some more, I'm not sure 
> how this can be
>  > >  > safely handled in standard JVMs either.
>  > > 
>  > > Exactly.  There aren't any significant differences between a
>  > > fully-compiled and a JIT environment from this point of 
> view.  So, if
>  > > they can do it, we can do it.
>  > 
>  > Is it possible that they're just giving the illusion of doing it
>  > properly?
> 
> It certainly is!
> 
>  > For instance, it's easy to imagine having a watchdog thread that
>  > periodically samples the various thread stack sizes and 
> forces large
>  > enough ones to throw errors.
> 
> It would have to be very fast.  
> 
> Perhaps they're doing stack probes at the entry point of every method.
> That would be cheap and easy.
> 
> Andrew.
>