MSVC hook function prologue

[Stefan Dösinger]

Hello,
After a rather long break due to other work I tried to revive my work on 
support for the function prologue used in Win32 API functions on Windows - a 
function prologue that some apps running in Wine expect.

This thread from January explains what I am trying to do:
http://gcc.gnu.org/ml/gcc/2009-01/msg00089.html

Essentially I want a function attrib that starts the function with this 
sequence, no matter what other parameters, code in the function, attributes 
or whatever are used:
8b ff          mov    %edi,%edi
55             push   %ebp
8b ec          mov    %esp,%ebp

I have attached the latest version of my patch for comments. It is mainly 
rebased against gcc changes that were made in the meantime. I also improved 
the REG_FRAME_RELATED_EXPR notes a bit, and only set it if the movs and pops 
are used for the frame pointer setup.

I also now know that I don't(or cannot) care about 64 bit right now. The 
windows apps currently do Windows API function hooking only in 32 bit, and 
there is no emerging scheme yet for hooking Win64 functions in the same way.

Currently I still have these problems:
*) There is apparently some plugin framework in the works. Can this 
functionality implemented as a plugin?

*) The way I read the msvc_prologue attribute seems wrong to me. I could read 
it directly in ix86_expand_prologue, but I got lost in the different trees 
gcc uses. I'm yet again trying to find this in the code and in the docs.

*) The code generated if no frame pointer is needed isn't pretty, but Wine 
will always need a frame pointer, so any optimization in that area won't get 
much test exposure.

*) The stack alignment code + msvc_prologue is used by Wine on osx though. 
Currently I pop %ebp after the 5 byte prologue, and the normal code recreates 
the frame pointer afterwards. My understanding is that I can avoid this by 
keeping the original frame pointer, but adjusting a lot of offsets after the 
alignment to find the function parameters and align the stack properly on 
calls. However, this is currently above my head.

*) What other changes are needed to get a functionality like this into 
mainline?

Thank you,
Stefan Dösinger
-------------- next part --------------
A non-text attachment was scrubbed...
Name: msvc-prologue.diff
Type: text/x-diff
Size: 6963 bytes
Desc: not available
URL: <https://gcc.gnu.org/pipermail/gcc/attachments/20090902/92b4e277/attachment.bin>