[PATCH] Add support for the Win32 hook prologue (try 7)

[Stefan Dösinger <stefan at codeweavers dot com>]

A number of Windows programs(Steam, Xfire and others) try to hook  
Win32 API
functions by replacing the first 5 bytes in the function. This causes
problems for Wine because the functions generated by gcc usually start  
with a
different opcode sequence than the one expected by these applications.

Starting with Windows XP SP2, Microsoft starts Win32 functions with this
sequence:

8b ff		mov %edi, %edi
55		push %ebp
8b ec	mov %esp, %ebp

The attached patch implements a function attribute that allows Wine to  
request
the same 5 bytes at the beginning of a function.

I tested the testuite on x86_64-pc-linux-gnu(and also in -m32 mode)

Changes from try 6:
the ms_hook_prologue attribute applies to the function, rather than  
its type. Changed the attribute lookup accordingly. However, I still  
allow the attribute for nested functions, there's no reason why this  
wouldn't work. (The case of static_chain and ms_hook_prologue can  
potentially be optimized)

Changes from try 5:
Fixed conflicts with static_chain patches, simple handling for
static_chain+ms_hook_prologue cases.
Improved changelog(hopefully correctly)

Changes from try 4:
Use SIMode instead of Pmode for the registers.

Changes from try 3:
Renamed the attribute name to ms_hook_prologue, since its not always  
generated
by msvc

Changes from try 2:
* Use dg-require-effective-target ilp32 instead of dg-options -m32 in  
the test
* More code layout fixes

Changes from the first try:
* Moved testcase to gcc.target/i386
* restructured ix86_handle_abi_attribute
* Changed HAVE_AS_IX86_SWAP handling as suggested by rth
* ix86_function_msvc_prologue returns a bool, removed fntype != NULL  
check
* fixed indentation in ix86_expand_prologue(hopefully correct now)

2009-09-23: Stefan Dösinger <stefan@codeweavers.com
       * config/i386/i386.md: Added vswapmov, a volative DIMode
          register->register mov with the swap suffix set

       * config/i386/i386.c (ix86_expand_prologue): Add a new function
       attribute ms_hook_prologue that starts functions with the same  
opcode
       sequence used in most Win32 API functions

       * config/i386/i386.c (ix86_function_ms_hook_prologue): New
       * config/i386/i386.c (ix86_handle_fndecl_attribute): New

       * gcc.target/i386/ms_hook_prologue.c: New testcase

       * configure.ac: Test for swap suffix support in as

Attachment: ms_hook_prologue.diff
Description: Binary data