mprotect reports ENOMEM under -O2

[Amittai Aviram]

Hi!  I have a modification of GCC that includes a call to mprotect from a signal handler.  Applications compiled with the altered GCC work fine (at least as far as I can tell) if I edit my GCC Makefile before compiling GCC so as to eliminate the default "-O2" flag.  If I restore the optimization, however, and recompile GCC and then the application under that GCC, the application fails:  mprotect reports ENOMEM.  Any ideas, either on this problem or on how I should go about investigating it?

A few background details:

The alteration is the beginning of a new implementation of OpenMP.  Among other differences, whereas the original implementation is built atop internal calls to the pthread machinery, this implementation is based on fork:   each "thread" is a separate process under the OS (Linux).  When each thread starts, it calls mprotect to set its heap and static data (BSS) segments to read-only, so that it can trap any first write to a page.  On the first write to a page, the signal handler records which page the thread has written to in a separate (read-write, of course) data structure, and resets the current page to read-write.  

The version of GCC I am altering is 4.5.2 .  I am running it and the target application under Ubuntu Linux 10.04.2 Server (x86_64) with kernel 2.6.32-server on a 64-bit AMD machine with 12 cores.

Thanks!

Amittai Aviram
PhD Student in Computer Science
Yale University
646 483 2639
amittai.aviram@yale.edu
http://www.amittai.com