Verification that taking address of a nested function always requires executable stack

Michael Meissner
Thu Nov 6 19:27:00 GMT 2008

On Wed, Nov 05, 2008 at 05:58:42PM -0800, Ian Lance Taylor wrote:
> "Kevin P. Fleming" <> writes:
> > We've been playing around with using nested functions recently; they've
> > got some interesting applications, but now that we've learned more about
> > how they are generally implemented, we're concerned because it requires
> > making portions (or all) of the stack executable.
> >
> > Is this *always* true? If so, should we be avoiding them completely so
> > as not to run into systems that disallow executable stack completely and
> > would not be able to run our code? If the stack is marked executable
> > (even partially), that seems to run some risk of buffer overflow code
> > execution anyway...
> The normal implementation of nested functions requires an executable
> stack when you take the address of a nested function.  If you use
> nested functions in ways that do not require taking their address
> (i.e., you simply call the functions), the stack does not need to be
> executable.
> On GNU/Linux gcc marks objects which require an executable stack, and
> the kernel will normally honor that marking when running the program.
> However, I believe that it is possible to configure GNU/Linux such
> that the stack is never permitted to be executable.
> If you expect to distribute your binaries to computers which you do
> not control, and which may be configured to disallow an executable
> stack, then you should indeed avoid using nested functions, or at
> least avoid taking their address.
> It would be possible to implement taking the address of a nested
> function differently, such that it does not require an executable
> stack, by using mmap and mprotect.  However, that has not been
> implemented, and there would be a relatively heavy runtime penalty.

I should note, that around 1990-1991, I tried rewrite the code in GCC that
assumes trampolines live on the stack (at the time the machine I was doing the
port for would have to do a system call to set up a trampoline, and since then
I've dealt with 2-3 machines that were impossible to write trampolines for).
Ultimately I gave up finding all of the hidden assumptions.  Maybe things have
changed in the meantime, but I suspect it may be not just an issue of adding a
few macros.

Michael Meissner, IBM
4 Technology Place Drive, MS 2203A, Westford, MA, 01886, USA

More information about the Gcc-help mailing list