Re: Help needed to modify prologue/epilogue expand methods...

[Ian Lance Taylor <iant at google dot com>]

Vaibhav Shrimali <vaibhav.shrimali@gmail.com> writes:

> i386.c : ix86_expand_prologue()
>
> /*************************************************************************************************/
> x = cfun->machine->force_align_arg_pointer;
> x = gen_frame_mem (Pmode, plus_constant (x, -8)) xor hard_frame_pointer_rtx;
> insn = emit_insn (gen_push (x));
> RTX_FRAME_RELATED_P (insn) = 1;
> /*************************************************************************************************/
>
> I am trying to push a copy of return address just below the return
> address and xor it with the "frame pointer". I wanted to know if the
> above code is correct for pushing the return address and encrypt it
> with the frame pointer.
> Now, in the epilogue I want to decrypt this copy of return address
> with the pushed frame pointer and verify if the two instances
> (original and decrypted) of the return address are same. This is the
> first time I am working on an approach that requires to make
> modifications to the ix86_expand_prologue/epilogue methods of the
> compiler. I also wanted to know how to read the value of encrypted
> return address in the ix86_expand_epilogue method, since then I need
> to decrypt it and compare it with the original return address pushed
> on the stack.

I don't know what you mean by the "xor" above, but otherwise that
looks more or less OK.  You will also have to account for the extra
word in the frame, in struct ix86_frame and ix86_compute_frame_layout.

I recommend that you take a look at the implementation of
-fstack-protector.  I think that is a better approach to this kind of
thing.

Ian