[PATCH] Add support for the Win32 hook prologue (try 5)

[Stefan Dösinger]

A number of Windows programs(Steam, Xfire and others) try to hook Win32 API 
functions by replacing the first 5 bytes in the function. This causes 
problems for Wine because the functions generated by gcc usually start with a 
different opcode sequence than the one expected by these applications.

Starting with Windows XP SP2, Microsoft starts Win32 functions with this 
sequence:

8b ff		mov %edi, %edi
55		push %ebp
8b ec	mov %esp, %ebp

The attached patch implements a function attribute that allows Wine to request 
the same 5 bytes at the beginning of a function.

I tested the testuite on x86_64-pc-linux-gnu.

Changes from try 4:
Use SIMode instead of Pmode for the registers.

Changes from try 3:
Renamed the attribute name to ms_hook_prologue, since its not always generated 
by msvc

Changes from try 2:
* Use dg-require-effective-target ilp32 instead of dg-options -m32 in the test
* More code layout fixes

Changes from the first try:
* Moved testcase to gcc.target/i386
* restructured ix86_handle_abi_attribute
* Changed HAVE_AS_IX86_SWAP handling as suggested by rth
* ix86_function_msvc_prologue returns a bool, removed fntype != NULL check
* fixed indentation in ix86_expand_prologue(hopefully correct now)

2009-09-09: Stefan Dösinger <stefan@codeweavers.com
        * config/i386/i386.c, config/i386/i386.md: Add a new function
        attribute ms_hook_prologue that starts functions with the same opcode 
        sequence used in most Win32 API functions

        * gcc.target/i386/ms_hook_prologue.c: New testcase

        * configure.ac: Test for swap suffix support in as
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ms_hook_prologue.diff
Type: text/x-diff
Size: 8875 bytes
Desc: not available
URL: <http://gcc.gnu.org/pipermail/gcc-patches/attachments/20090917/dff60f10/attachment.bin>