[patch] libmudflap, freebsd malloc issue

[James E Wilson <wilson at specifix dot com>]

This addresses a problem I originally reported back in February.

On an x86 FreeBSD 4.10 system, trying to compile gcc with -fmudflap
results in the gengtype program failing with an complaint about a
recursive malloc call.

The FreeBSD malloc uses mmap to allocate a table used to keep track of
all pages allocated on the heap.  Pages are 4K, and the entries are 4
bytes each, so we can track 4MB (4KB / 4 * 4KB) of allocated data in
each mmap allocated page.  The initial page is allocated in a safe
context.  However, if this page table map needs to be expanded in
response to an instrumented malloc call, then we have a problem.  The
program calls the malloc wrapper, which calls the real malloc, which
calls the mmap wrapper, which calls the real malloc to register the
data.  Now we have a recursive malloc call, and that triggers a fatal
signal inside the FreeBSD library.  This is trivial to reproduce with a
program that allocates 4MB with a single malloc call.  Because
alpha-freebsd has 8KB pages, we need to allocate 16MB to reproduce the
problem there.

The same problem occurs with free calling munmap.

This patch fixes the problem by adding in_malloc to the existing
mf_state enum.  We set this state before malloc/free wrappers call the
real routines, and reset it afterward.  In the wrappers, we check for
the in_malloc state, and in that case we just directly call the real
routines.

This has been tested with a C and C++ bootstrap and make check on an
x86-freebsd4.10 system.  There were no regressions.  The new testcase
fails without the patch, and works with the patch.

There are a few issues here people may want to discuss.  I did not make
this code target dependent, even though linux doesn't need it.  I'm not
sure how useful the new testcase is.  It won't work for many embedded
targets for instance because of the large memory allocation, but since
libmudflap is only built for FreeBSD and Linux this shouldn't be a
problem.

I'm waiting for discussion before trying to check this in.
-- 
Jim Wilson, GNU Tools Support, http://www.specifix.com

Attachment: patch.malloc.protect
Description: Text document