[Patch/i386]: -mrtd vs system library declarations

[Danny Smith <danny_r_smith_2001 at yahoo dot co dot nz>]

The -mrtd switch for i386 targets applies the __stdcall__
(or PASCAL) calling convention (callee responsible for
popping stack) as the default to all function definitions or
declarations. Although this is sometime useful in fortran or
(gasp) VisualBasic, it causes problems with inclusion of system
headers for runtime libraries.  With -mrtd, calling
code believes that the declared standard runtime libary functions
were compiled as stdcall functions, unless they are
explicitly marked with __attribute__((cdecl)). This leads to
stack corruption.

Here is an example (assuming <math.h> has not explicitly
marked log() with __attribute__((cdecl)) )

/* mrtd.c */
#include <math.h>
void foo1(double z)
{
  double x = log (z);
}

gcc -S -mrtd.c

produce this:

	.file	"mrtd.c"
	.text
.globl _foo1
	.def	_foo1;	.scl	2;	.type	32;	.endef
_foo1:
	pushl	%ebp
	movl	%esp, %ebp
	subl	$24, %esp
	fldl	8(%ebp)
	fstpl	-8(%ebp)
	fldl	-8(%ebp)
	fstpl	(%esp)
	call	_log
	subl	$8, %esp  # <<< assumes callee  pops the stack 
	fstpl	-16(%ebp)
	leave
	ret	$8
	.def	_log;	.scl	3;	.type	32;	.endef

OK, a tedious but simple fix for above is to explicitly mark
all C runtime library funtions as "cdecl". In fact, mingw
has done this. But we still have problems with other system
libraries -- notably libstdc++. Maintaining local patches to
mark libstdc++ library functions as "cdecl" is not my idea
of fun.  And then there is libiconv and libintl, for cygwin there is
newlib, and so on.

The following patch to config/i386/i386.c provides
protection for system library headers, by ignoring the -mrtd
switch for those headers in system include directories, or
marked with #pragma GCC system_header. If system library
functions really do want "stdcall" convention, they need to
mark the function explicitly.

Tested on i686-pc-mingw32/mainline GCC.

As an alternative, I believe there should be a strong
warning in doc's (and in the preprocessor itself) that the
-mrtd switch should not be used in code that includes system
headers.

ChangeLog

2003-07-22  Danny Smith  <dannysmith@users.sourceforge.net>

	* config/i386/i386.c (ix86_return_pops_args): Ignore
	-mrtd for system headers. 

Index: i386.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/config/i386/i386.c,v
retrieving revision 1.589
diff -c -3 -p -r1.589 i386.c
*** i386.c	15 Jul 2003 13:42:39 -0000	1.589
--- i386.c	22 Jul 2003 07:50:33 -0000
*************** ix86_fntype_regparm (tree type)
*** 1699,1710 ****
     standard Unix calling sequences.  If the option is not selected,
     the caller must always pop the args.
  
     The attribute stdcall is equivalent to RTD on a per module basis.  */
  
  int
  ix86_return_pops_args (tree fundecl, tree funtype, int size)
  {
!   int rtd = TARGET_RTD && (!fundecl || TREE_CODE (fundecl) != IDENTIFIER_NODE);
  
      /* Cdecl functions override -mrtd, and never pop the stack.  */
    if (! lookup_attribute ("cdecl", TYPE_ATTRIBUTES (funtype))) {
--- 1699,1716 ----
     standard Unix calling sequences.  If the option is not selected,
     the caller must always pop the args.
  
+    Ignore the -mrtd switch for system headers, otherwise system library
+    functions will be expected to pop the stack unless explicitly
+    marked as "cdecl".
+ 
     The attribute stdcall is equivalent to RTD on a per module basis.  */
  
  int
  ix86_return_pops_args (tree fundecl, tree funtype, int size)
  {
!   int rtd = TARGET_RTD
! 	    && (!fundecl || (TREE_CODE (fundecl) != IDENTIFIER_NODE
! 			     && !DECL_IN_SYSTEM_HEADER (fundecl)));
  
      /* Cdecl functions override -mrtd, and never pop the stack.  */
    if (! lookup_attribute ("cdecl", TYPE_ATTRIBUTES (funtype))) {

http://mobile.yahoo.com.au - Yahoo! Mobile
- Check & compose your email via SMS on your Telstra or Vodafone mobile.