[Bug target/12865] New: `mprotect' call to make trampoline executable may fail

["hebisch at math dot uni dot wroc dot pl" <gcc-bugzilla at gcc dot gnu dot org>]

PLEASE REPLY TO gcc-bugzilla@gcc.gnu.org ONLY, *NOT* gcc-bugs@gcc.gnu.org.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12865

           Summary: `mprotect' call to make trampoline executable may fail
           Product: gcc
           Version: 3.3.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: target
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: hebisch at math dot uni dot wroc dot pl
                CC: gcc-bugs at gcc dot gnu dot org
 GCC build triplet: sparc-sun-solaris2.6
  GCC host triplet: sparc-sun-solaris2.6
GCC target triplet: sparc-sun-solaris2.6

On Sparc Solaris 2.6 the program below produces spurious
error message:
mprotect of trampoline code: Not enough space

The program should just print "OK". When compiled with '-O2' option
the error message appeared. When I used 8 or 9 letter executable name
the problem disappeard, however the error occurs with the name
'a.out' or longer names.

----------------------- <cut here >--------------------------------
#include <stdio.h>
The program should just print "OK". When compiled with '-O2' option#include <stdlib.h>
the error message appeared.#include <math.h>

void manorboy (void)
{
const
  int correct[1100] = {1, 0, -2, 0, 1, 0, 1, -1, -10, -30, -67};

  int i;
  double x1 (void) {return 1;}
  double x2 (void) {return -1;}
  double x3 (void) {return -1;}
  double x4 (void) {return 1;}
  double x5 (void) {return 0;}
  typedef double pfun(void);
  double a (int k, pfun x1, pfun x2, pfun x3, pfun x4, pfun x5)
  {
    double b (void)
    {
        k = k - 1;
        return a (k, b, x1, x2, x3, x4 );
    }
    if (k <= 0) {
      return x4 () + x5 ();
    } else {
      return b ();
    }
  }
  for (i=0; i<=10; i++) {
        if (fabs(a( i, x1, x2, x3, x4, x5 ) - correct [i])>0.1) {
                fprintf(stderr, "Failed\n");
                exit(1);
        }
  }
  fprintf(stderr, "OK\n");
}

int main(void)
{
  manorboy ();
  return EXIT_SUCCESS;
}

The problem actually was discoverd with GNU Pascal, were nested
functions are used frequently. My findigs are:
-- on Solaris mprotect is used to make tramplines executable
-- mprotect may fail if given unmapped adress
-- gcc can make rather lage adjastments to the stack
   pointer
-- `mprotect' was called before any stack access

My hypotesis is that Solaris kernel extended stack only on
actual memory access, producing error on `mprotect'
The patch below fixes the problem:
--- gcc-3.3.1.orig/gcc/config/sparc/sparc.c     Wed Jul 16 07:19:33 2003
+++ gcc-3.3.1/gcc/config/sparc/sparc.c  Tue Oct 14 01:04:01 2003
@@ -6777,10 +6777,6 @@
     SETHI i,r  = 00rr rrr1 00ii iiii iiii iiii iiii iiii
     JMPL r+i,d = 10dd ddd1 1100 0rrr rr1i iiii iiii iiii
    */
-#ifdef TRANSFER_FROM_TRAMPOLINE
-  emit_library_call (gen_rtx (SYMBOL_REF, Pmode, "__enable_execute_stack"),
-                     LCT_NORMAL, VOIDmode, 1, tramp, Pmode);
-#endif

   emit_move_insn
     (gen_rtx_MEM (SImode, plus_constant (tramp, 0)),
@@ -6811,6 +6807,13 @@
                   expand_and (SImode, cxt, GEN_INT (0x3ff), NULL_RTX),
                   GEN_INT (trunc_int_for_mode (0x8410a000, SImode)),
                   NULL_RTX, 1, OPTAB_DIRECT));
+
+  /* Call `__enable_execute_stack' after stack is extended, otherwise
+     `mprotect' may fail. (hebisch@math.uni.wroc.pl) */
+#ifdef TRANSFER_FROM_TRAMPOLINE
+  emit_library_call (gen_rtx (SYMBOL_REF, Pmode, "__enable_execute_stack"),
+                     LCT_NORMAL, VOIDmode, 1, tramp, Pmode);
+#endif

   /* On UltraSPARC a flush flushes an entire cache line.  The trampoline is
      aligned on a 16 byte boundary so one flush clears it all.  */
@@ -6829,11 +6832,6 @@
 sparc64_initialize_trampoline (tramp, fnaddr, cxt)
      rtx tramp, fnaddr, cxt;
 {
-#ifdef TRANSFER_FROM_TRAMPOLINE
-  emit_library_call (gen_rtx (SYMBOL_REF, Pmode, "__enable_execute_stack"),
-                     LCT_NORMAL, VOIDmode, 1, tramp, Pmode);
-#endif
-
   /*
        rd      %pc, %g1
        ldx     [%g1+24], %g5
@@ -6852,8 +6850,15 @@
                  GEN_INT (trunc_int_for_mode (0xca586010, SImode)));
   emit_move_insn (gen_rtx_MEM (DImode, plus_constant (tramp, 16)), cxt);
   emit_move_insn (gen_rtx_MEM (DImode, plus_constant (tramp, 24)), fnaddr);
-  emit_insn (gen_flushdi (validize_mem (gen_rtx_MEM (DImode, tramp))));

+  /* Call `__enable_execute_stack' after stack is extended, otherwise
+     `mprotect' may fail. (hebisch@math.uni.wroc.pl) */
+#ifdef TRANSFER_FROM_TRAMPOLINE
+  emit_library_call (gen_rtx (SYMBOL_REF, Pmode, "__enable_execute_stack"),
+                     LCT_NORMAL, VOIDmode, 1, tramp, Pmode);
+#endif
+
+  emit_insn (gen_flushdi (validize_mem (gen_rtx_MEM (DImode, tramp))));
   if (sparc_cpu != PROCESSOR_ULTRASPARC
       && sparc_cpu != PROCESSOR_ULTRASPARC3)
     emit_insn (gen_flushdi (validize_mem (gen_rtx_MEM (DImode, plus_constant (tramp, 8)))));

I tested the patch with gcc-3.3.1, but it applies to gcc-3.3-20031027
and gcc-3.4-20031029.