x86_64 calling conventions and stack frames

[Bob Plantz]

On 12/23/2011 11:37 PM, Amittai Aviram wrote:
> I am trying to make sense out of the executable code that GCC (4.4.3) is generating for an x86_64 machine running under Ubuntu Linux.  In particular, I don't understand how the code keeps track of stack frames.  In the old days, in 32-bit code, I am accustomed to seeing this as a "prologue" in just about every function:
>
> push %ebp
> movl %esp, %ebp
>
> Then, at the end of the function, there would either be
>
> sub $xx, %esp   # Where xx is a number based on GCC's accounting.
> pop %ebp
> ret
>
> or simply
>
> leave
> ret
>
> which accomplishes the same thing:
> - Set the Stack Pointer to the top of the current frame, just below the return address
> - Restore the old Frame Pointer value.
>
> In 64-bit code, as I see it through an objdump disassembly, many functions do not follow this convention--they do not push %rbp and then save %rsp to %rbp, How does a debugger like GDB build a backtrace?

Are you looking at leaf functions? They are allowed to use the "red 
zone" as their stack frame, with rsp as the reference point. Since there 
are twice as many registers in 64-bit mode, you will find many fewer 
pushes and pops. Most arguments (the first six integer) are passed in 
registers and simply moved to the stack frame for safe keeping.

> x86-64
>
> My real goal here to is to try to figure out a reasonable address to consider as the top (highest address) of the user stack when execution reaches the start of a function further into the program, where perhaps the Stack Pointer has moved down.  I had at first thought that I could use the old backtrace method: chasing saved Frame Pointer values until the value saved is 0--then, the next one after that can count as the highest practical value.  Now, I don't know how to get the equivalent address in 64-bit code.

If you don't have it, you should get a copy of "System V Application 
Binary Interface AMD64 Architecture Processor Supplement." It was 
available at www.x86-64.org, but I can't seem to connect to them right 
now. The latest version I have is 0.99 (Dec. 7, 2007). It describes the 
red zone, etc.

--Bob