stack overflow

[Godmar Back]

> 
> Godmar Back <gback@cs.utah.edu> writes:
> 
> > In April, we've had a brief discussion on this list about stack 
> > overflow detection.  I noticed that it's still not implemented
> > in the recent snapshot I compiled.
> > 
> > Has there been any design or implementation since?  I've grepped
> > the mailing list archives, but couldn't find anything.
> 
> Well, check *does* have a -fstack-check option, which does
>     "Emit code to check for stack overflow; also may cause large objects
>      to be allocated dynamically."
> Have you tried it?

Hmmm, it seems a though this option generates a memory access to a page
below the stack pointer at every proc entry.  On the x86, for instance:

        leal    -4392(%esp), %eax
	movl    $0, (%eax)

This seems as though you're going the hardware route...
I take it then that you're convinced you can implement it
portably.  Hmmm, maybe I should just quit discussing things and
simply try it and see just how hairy it would be.

My Overflow test though still fails with libgcj---presumably it's not 
supported yet in libgcj?  'rgrep -r StackOverflow sourcetree' doesn't show 
anything, and if I look at the catch_segv handler it seems as though every 
SIGSEGV is interpreted as a nullpointer exception.

	- Godmar

ps.:
This is what the latte say about hardware stackoverflow detection on the
sparc.  However, this looks like all they'd have to do is to use 
sigaltstack(2) (?)

http://latte.snu.ac.kr/src/BUGS

    Bugs in the runtime
    ===================

    Does not detect stack overflows.  The only practical way to do this is
    to check for stack overflows at each method invocation, which is
    extremely detrimental to performance.  This is mainly because Java
    methods use the native calling convention for the calling convention,
    precluding the use of mprotect() and signal handlers.  The reason for
    this is that on the SPARC, when using read only pages to detect stack
    overflow, the trap detecting the overflow occurs when register windows
    are saved to a read only page, then the trap tries to save the
    register windows again, which raises a trap again, which is not
    caught.  At least this is what is thought to be preventing mprotect()
    and signal handlers from being able to catch stack overflows.  If
    traps don't save register windows, then the SAVE instruction done when
    entering the signal handler would be the cause, so it might be a
    worthwhile attempt to use a separate entry point to the trap handler
    which switches the stack to some valid page before doing any saves.
    If even this doesn't work, we're screwed, unless someone turns up
    another solution or the calling conventions for Java methods are
    changed.

	- Godmar

pps.:
Btw, when I type 'gcj -fwhatever Overflow.java' --- gcj doesn't complain.
This is so wrong.  It's like autoconf generated configure that also don't
complain if you give an option that's not supported.  I've wasted hours
and hours spotting mistakes such as giving "--with-libraries=" when configure 
wanted "--with-library-dir=" or vice versa.