addendum to gcc/config/i386/i386.c patch

[Marcus Meissner <marcus at jet dot franken dot de>]

Hi.

As Addendum to my patch the 'misbehaviour' it tries to fix.
[I though it was obvious, but it (probably) isn't. Sorry.]

The bug is only relevant on the i386 architecture.

x86 may use two different calling conventions, which are usually named
stdcall and cdecl. 

- stdcall: the called function removes the passed arguments from the stack
  (using an lret n instruction).
- cdecl: the caller removes the passed arguments from the stack after the
  return (using add esp, ... or whatever).

Operating Systems usually use just one of those conventions. UNIX like
OSes use "cdecl" while Windows uses "stdcall" in its Core APIs (some 
APIs do use "cdecl" too).

gcc contains support for those two attributes by using the i386 architecture
specific attributes "cdecl" and "stdcall" which modify the return behaviour.
The compiler is built with one of these as default, but the behaviour can be
changed using the compiler switch -mrtd (?) or __attribute__().

The problem now is, when a function is declared with a different attribute
than its definition, there is NO WARNING and NO ERROR printed, just the
resulting code is broken. I have included a small example below.

-----------------------------xx.c-------------------------------------
int foo(int x,int y);

int main(void) {
	foo(1,2);
}

int __attribute__((__stdcall__)) foo(int x,int y) {
	return x+y;
}
------------------------------------------------------------------------

$ gcc -fomit-frame-pointer -O4 -c  xx.c
(-fomit-frame-pointer -O4 just to make the assembly code smaller).
This compiles without any warnings.
$ objdump -d xx.o
|xx.o:     file format elf32-i386
|Disassembly of section .text:
|00000000 <main>:
|   0:	6a 02          	pushl  $0x2
|   2:	6a 01          	pushl  $0x1
|   4:	e8 07 00 00 00 	call   10 <foo>
|   9:	83 c4 08       	addl   $0x8,%esp	; cdecl like argument removal
|   c:	c3             	ret    
|00000010 <foo>:
|  10:	8b 44 24 04    	movl   0x4(%esp,1),%eax
|  14:	03 44 24 08    	addl   0x8(%esp,1),%eax
|  18:	c2 08 00       	ret    $0x8		; stdcall like argument removal

Now, calling foo from main removes the arguments from the stack TWICE,
so messing up %esp. This may lead (depending on the use of %esp) to stack
corruptions.

"gcc -o xx xx.c ; ./xx" does not result in an error, since %esp is not
used after the call to foo since it is restored from %ebp at the end of main.

"gcc -fomit-frame-pointer -o xx xx.c; ./xx" results in "Segmentation fault",
for main() uses the wrong return address.

This misbehaviour has shown up in the free Windows Emulator WINE. I personally
spent approx 10 full hours looking for the reason of mysterious stack
corruptions originating from attribute-vise mismatched definitions/declarations.

Since this is my first ever visit to the gcc source (I spent 3 hours or so
finding the correct place, the meaning of the TREE stuff, and hacking
up a working patch) the patch is unfortunately suboptimal.

The patch is in http://www.cygnus.com/ml/egcs-patches/1998-Oct/0607.html.

And. Do I *really* need to fill out the Copyright Disclaimer? I thought, after
reading the "how to submit stuff to egcs" document, that this (as a "small
change") does not need a disclaimer?

Ciao, Marcus