Bug 12865 - [3.3/3.4 regression] `mprotect' call to make trampoline executable may fail
Summary: [3.3/3.4 regression] `mprotect' call to make trampoline executable may fail
Status: RESOLVED FIXED
Alias: None
Product: gcc
Classification: Unclassified
Component: target (show other bugs)
Version: 3.3.1
: P2 critical
Target Milestone: 3.3.3
Assignee: Eric Botcazou
URL:
Keywords: patch
Depends on:
Blocks:
 
Reported: 2003-11-01 02:26 UTC by hebisch
Modified: 2004-01-17 04:22 UTC (History)
1 user (show)

See Also:
Host: sparc-sun-solaris2.6
Target: sparc-sun-solaris2.6
Build: sparc-sun-solaris2.6
Known to work:
Known to fail:
Last reconfirmed: 2003-11-01 07:19:36


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description hebisch 2003-11-01 02:26:04 UTC
On Sparc Solaris 2.6 the program below produces spurious
error message:
mprotect of trampoline code: Not enough space

The program should just print "OK". When compiled with '-O2' option
the error message appeared. When I used 8 or 9 letter executable name
the problem disappeard, however the error occurs with the name
'a.out' or longer names.

----------------------- <cut here >--------------------------------
#include <stdio.h>
The program should just print "OK". When compiled with '-O2' option#include <stdlib.h>
the error message appeared.#include <math.h>

void manorboy (void)
{
const
  int correct[1100] = {1, 0, -2, 0, 1, 0, 1, -1, -10, -30, -67};

  int i;
  double x1 (void) {return 1;}
  double x2 (void) {return -1;}
  double x3 (void) {return -1;}
  double x4 (void) {return 1;}
  double x5 (void) {return 0;}
  typedef double pfun(void);
  double a (int k, pfun x1, pfun x2, pfun x3, pfun x4, pfun x5)
  {
    double b (void)
    {
        k = k - 1;
        return a (k, b, x1, x2, x3, x4 );
    }
    if (k <= 0) {
      return x4 () + x5 ();
    } else {
      return b ();
    }
  }
  for (i=0; i<=10; i++) {
        if (fabs(a( i, x1, x2, x3, x4, x5 ) - correct [i])>0.1) {
                fprintf(stderr, "Failed\n");
                exit(1);
        }
  }
  fprintf(stderr, "OK\n");
}

int main(void)
{
  manorboy ();
  return EXIT_SUCCESS;
}

The problem actually was discoverd with GNU Pascal, were nested
functions are used frequently. My findigs are:
-- on Solaris mprotect is used to make tramplines executable
-- mprotect may fail if given unmapped adress
-- gcc can make rather lage adjastments to the stack
   pointer
-- `mprotect' was called before any stack access

My hypotesis is that Solaris kernel extended stack only on
actual memory access, producing error on `mprotect'
The patch below fixes the problem:
--- gcc-3.3.1.orig/gcc/config/sparc/sparc.c     Wed Jul 16 07:19:33 2003
+++ gcc-3.3.1/gcc/config/sparc/sparc.c  Tue Oct 14 01:04:01 2003
@@ -6777,10 +6777,6 @@
     SETHI i,r  = 00rr rrr1 00ii iiii iiii iiii iiii iiii
     JMPL r+i,d = 10dd ddd1 1100 0rrr rr1i iiii iiii iiii
    */
-#ifdef TRANSFER_FROM_TRAMPOLINE
-  emit_library_call (gen_rtx (SYMBOL_REF, Pmode, "__enable_execute_stack"),
-                     LCT_NORMAL, VOIDmode, 1, tramp, Pmode);
-#endif

   emit_move_insn
     (gen_rtx_MEM (SImode, plus_constant (tramp, 0)),
@@ -6811,6 +6807,13 @@
                   expand_and (SImode, cxt, GEN_INT (0x3ff), NULL_RTX),
                   GEN_INT (trunc_int_for_mode (0x8410a000, SImode)),
                   NULL_RTX, 1, OPTAB_DIRECT));
+
+  /* Call `__enable_execute_stack' after stack is extended, otherwise
+     `mprotect' may fail. (hebisch@math.uni.wroc.pl) */
+#ifdef TRANSFER_FROM_TRAMPOLINE
+  emit_library_call (gen_rtx (SYMBOL_REF, Pmode, "__enable_execute_stack"),
+                     LCT_NORMAL, VOIDmode, 1, tramp, Pmode);
+#endif

   /* On UltraSPARC a flush flushes an entire cache line.  The trampoline is
      aligned on a 16 byte boundary so one flush clears it all.  */
@@ -6829,11 +6832,6 @@
 sparc64_initialize_trampoline (tramp, fnaddr, cxt)
      rtx tramp, fnaddr, cxt;
 {
-#ifdef TRANSFER_FROM_TRAMPOLINE
-  emit_library_call (gen_rtx (SYMBOL_REF, Pmode, "__enable_execute_stack"),
-                     LCT_NORMAL, VOIDmode, 1, tramp, Pmode);
-#endif
-
   /*
        rd      %pc, %g1
        ldx     [%g1+24], %g5
@@ -6852,8 +6850,15 @@
                  GEN_INT (trunc_int_for_mode (0xca586010, SImode)));
   emit_move_insn (gen_rtx_MEM (DImode, plus_constant (tramp, 16)), cxt);
   emit_move_insn (gen_rtx_MEM (DImode, plus_constant (tramp, 24)), fnaddr);
-  emit_insn (gen_flushdi (validize_mem (gen_rtx_MEM (DImode, tramp))));

+  /* Call `__enable_execute_stack' after stack is extended, otherwise
+     `mprotect' may fail. (hebisch@math.uni.wroc.pl) */
+#ifdef TRANSFER_FROM_TRAMPOLINE
+  emit_library_call (gen_rtx (SYMBOL_REF, Pmode, "__enable_execute_stack"),
+                     LCT_NORMAL, VOIDmode, 1, tramp, Pmode);
+#endif
+
+  emit_insn (gen_flushdi (validize_mem (gen_rtx_MEM (DImode, tramp))));
   if (sparc_cpu != PROCESSOR_ULTRASPARC
       && sparc_cpu != PROCESSOR_ULTRASPARC3)
     emit_insn (gen_flushdi (validize_mem (gen_rtx_MEM (DImode, plus_constant (tramp, 8)))));

I tested the patch with gcc-3.3.1, but it applies to gcc-3.3-20031027
and gcc-3.4-20031029.
Comment 1 Eric Botcazou 2003-11-01 07:19:35 UTC
Confirmed on Solaris 2.6 and 64-bit Solaris 7, 8 and 9. Solaris 2.5.1 and 32-bit
Solaris 7, 8 and 9 appear to be immune.
Comment 2 Eric Botcazou 2003-11-01 07:23:16 UTC
Regression from GCC 2.95.3 on Solaris 2.6.
Comment 3 Eric Botcazou 2003-11-01 07:23:48 UTC
I'll take care of it.
Comment 4 hebisch 2003-11-01 13:26:53 UTC
I already have posted the patch also to gcc-patches:
http://gcc.gnu.org/ml/gcc-patches/2003-11/msg00009.html
Comment 5 GCC Commits 2003-11-10 08:11:50 UTC
Subject: Bug 12865

CVSROOT:	/cvs/gcc
Module name:	gcc
Changes by:	ebotcazou@gcc.gnu.org	2003-11-10 08:11:47

Modified files:
	gcc            : ChangeLog 
	gcc/config/sparc: sparc.c 
	gcc/testsuite  : ChangeLog 
Added files:
	gcc/testsuite/gcc.dg: trampoline-1.c 

Log message:
	PR target/12865
	* config/sparc/sparc.c (sparc_initialize_trampoline): Call
	__enable_execute_stack only after writing onto the stack.
	(sparc64_initialize_trampoline): Likewise.

Patches:
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ChangeLog.diff?cvsroot=gcc&r1=2.1711&r2=2.1712
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/config/sparc/sparc.c.diff?cvsroot=gcc&r1=1.268&r2=1.269
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/testsuite/ChangeLog.diff?cvsroot=gcc&r1=1.3178&r2=1.3179
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/testsuite/gcc.dg/trampoline-1.c.diff?cvsroot=gcc&r1=NONE&r2=1.1

Comment 6 GCC Commits 2003-11-10 08:17:10 UTC
Subject: Bug 12865

CVSROOT:	/cvs/gcc
Module name:	gcc
Branch: 	gcc-3_3-branch
Changes by:	ebotcazou@gcc.gnu.org	2003-11-10 08:17:00

Modified files:
	gcc            : ChangeLog 
	gcc/config/sparc: sparc.c 
	gcc/testsuite  : ChangeLog 
Added files:
	gcc/testsuite/gcc.dg: trampoline-1.c 

Log message:
	PR target/12865
	* config/sparc/sparc.c (sparc_initialize_trampoline): Call
	__enable_execute_stack only after writing onto the stack.
	(sparc64_initialize_trampoline): Likewise.

Patches:
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ChangeLog.diff?cvsroot=gcc&only_with_tag=gcc-3_3-branch&r1=1.16114.2.802&r2=1.16114.2.803
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/config/sparc/sparc.c.diff?cvsroot=gcc&only_with_tag=gcc-3_3-branch&r1=1.233.4.6&r2=1.233.4.7
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/testsuite/ChangeLog.diff?cvsroot=gcc&only_with_tag=gcc-3_3-branch&r1=1.2261.2.315&r2=1.2261.2.316
http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/testsuite/gcc.dg/trampoline-1.c.diff?cvsroot=gcc&only_with_tag=gcc-3_3-branch&r1=NONE&r2=1.1.2.1

Comment 7 Eric Botcazou 2003-11-10 08:20:22 UTC
See http://gcc.gnu.org/ml/gcc-patches/2003-11/msg00350.html