Stack Checking on x86

[Marc E. Fiuczynski]

Hi,

I want to develop a discontiguous stack for the Linux/x86 kernel such that I
can use, say, 512byte stacks and upon overflow jump to a new stack.  I know,
weird thing to do, but I have my reasons.

To detect stack overflow, my first implementation leveraged the profile
support (i.e., -p) and a custom mcount implementation.  Unfortunately, this
has the negative side effect that the call to mcount is made _after_ the
stack is adjusted for the local frame --- on x86 the ret addr. push onto the
stack as part of the call instruction will corrupt system state.  Moreover,
it only works for functions with no args --- rather rare in most programs,
no?! :)

So I thought I'd leverage the Stack Checking support that gcc has
(i.e., -fstack-check). Was reading the online docs on Stack Checking
(http://gcc.gnu.org/onlinedocs/gccint/Stack-Checking.html) and been peeking
through the code in gcc/config/i386/*. Are there any examples of defining
the check_stack pattern for x86?

Finally, I am considering of modifying TARGET_ASM_FUNCTION_PROLOGUE, but
that thing does not seem to be defined in config/i386.  I did see was
ix86_expand_prologue in config/i386/i386.c.  Is that the right spot to do
this type of hacking?

Thanks,
Marc