Re: Inline asm function calls + red zone

[Ian Lance Taylor <iant at google dot com>]

Jason Garrett-Glaser <jason@x264.com> writes:

> From talking with others, there appears to be a problem with function
> calls in inline asm on x86_64: the call clobbers the first 8 bytes of
> the stack red zone, which GCC is allowed to use for other data in the
> function.  This is a problem even if the function being called doesn't
> use the stack, because "call" itself does use the stack.  Besides the
> extremely hacky sequence of:
>
> sub esp, 128
> call func
> add esp, 128
>
> Is there a way to tell gcc not to use the red zone in a function, or
> that part of the red zone is going to be clobbered by the inline
> assembly code?

In general making function calls from asm code is not supported, and
this is one of the reasons why that is so.  There are many targets for
which gcc optimizes leaf functions differently from non-leaf functions.
An asm with a function call turns a leaf function into a non-leaf
function, but gcc doesn't know that that is happening.  This causes
things to break.

You can work around this specific issue by using -mno-red-zone when you
compile the file containing the asm.  I can't guarantee that you won't
run into other issues.

Ian