question about the usage of DECL_INCOMING_RTL in a stack-smashing protector

[Hiroaki Etoh]

Jim Wilson <wilson@specifixinc.com> wrote on 2004/02/05 18:57:13:

> Hiroaki Etoh wrote:
> > I'd like to know issues about setting a pseudo register to 
> > DECL_INCOMING_RTL.
> 
> DECL_INCOMING_RTL is the location where the caller stored the reg. 
> Since the caller couldn't have put it in a pseudo-reg, this has never 
> been supported.  Just taking a quick look though, I don't see any 
> obvious reason why this won't work.

I don't want to change the location where the caller stored the reg.  I 
want to change the use of function arguments.  I copied arguments to new 
local variables  and changed the use of arguments to the use of new 
variables.   I'd like to do it because pointer arguments should be 
protected from a stack smashing attack.  I prepared a web paper for this 
protection method.  please see the page:
http://www.research.ibm.com/trl/projects/security/ssp/node4.html#SECTION00042000000000000000

As described in the paper, the protector finds every instructions which 
uses a function argument and changes them to use the new  location of the 
argument.  It works fine, but I found that there is a limitation to use 
gdb.  gdb shows the content of the original function argument, not the 
content of the new location.  I examined the gcc source code and found 
that the location for the debug information is gotten from 
DECL_INCOMING_RTL.   So, I set the new location to DECL_INCOMING_RTL and 
have examined many programs that gdb successfully checks the function 
argument.


> Gcc already copies arguments into pseudos, this is DECL_RTL.  It isn't 
> clear why you need another one.

Yes, gcc copies arguments into pseudos when it is used in an optimization 
option.  The reason why I need another one is for preventing pseudo 
elimination, done by register optimizer.  It eliminates pseudos by 
replacing them with the equivalent hard register or memory.   The 
duplicated function argument should not be eliminated because the original 
location is vulnerable from an attack. 

  I can't find out which source codes eliminate the pseudos for the 
purpose of optimization.  So, I introduced a new pseudo and stop register 
propagation based on its equivalence when the new pseudo is found.