Re: Add support for the Win32 hook prologue (try 3)

[Stefan Dösinger <stefan at codeweavers dot com>]

Am Friday 11 September 2009 23:53:57 schrieb Richard Henderson:
> On 09/11/2009 01:51 PM, Stefan Dösinger wrote:
> >      if (/* ix86_function_ms_hook_prologue (TREE_TYPE (decl)) */ 1) {
> >          gen_nop(); gen_nop(); gen_nop(); gen_nop(); gen_nop();
>
> This certainly won't work.  You need to output to FILE directly with fputs.
Figured that out a minute ago after the compile was done :-)

> > #define IX86_ADD_HOOK_NOPS ix86_hook_nops;
>
> Not IX86 anything, since you'll be putting an empty copy of this hook
> into non-i386 files too.  Perhaps
True.

But still it seems nicer to integrate this with the function that generates 
the padding. My current implementation that is based on 
ASM_DECLARE_FUNCTION_NAME generates code like this:

// .align here
00000000 <test-0x5>:
   0:   90                      nop
   1:   90                      nop
   2:   90                      nop
   3:   90                      nop
   4:   90                      nop

00000005 <test>:
   5:   8b ff                   mov    %edi,%edi
   7:   55                      push   %ebp
   8:   8b ec                   mov    %esp,%ebp
etc

Obviously that's not too nice when it comes to the alignment of the function 
entrypoint. On the plus side, I have full control over the code before the 
function

I guess one of the mentioned suggestions that adds this to the alignment code 
would produce something like this:

12345678 <test-0x5>:
   90                      nop
   90                      nop
   90                      nop
   90                      nop
   90                      nop
.align 4, 0xcc
12345680 <test>:
   5:   8b ff                   mov    %edi,%edi
   7:   55                      push   %ebp
   8:   8b ec                   mov    %esp,%ebp

The function start is at an aligned address, but as discussed earlier, the 
assembler controls which type of NOP is generated. But since other people 
suggested that MSFT sometimes uses 0xCC that's ok I guess.
(I personally cannot disassemble Windows DLLs since it might taint my Wine 
work. At least a lawyer could argue that way)

What's the best way to do this? Add a new macro ASM_OUTPUT_FUNCTION_ALIGN as 
you suggested that adds the 5 nops(if needed), then does the normal work of 
ASM_OUTPUT_ALIGN?

> > Also, it I guess its a good idea to split the in-function two byte nop
> > and the nops before the function into two different patches. (Maybe the
> > outer nops first, then the things inside the function)
>
> How about the internal patch first, since that is, for the most part, done.
My consideration was that apps micht see the 8b ff and try to overwrite other 
code, but I guess as long as Wine doesn't start using it before the 5 nops 
are there as well there's not too much danger. I'll send the patch.