I have found what appears to be a pretty serious issue with the new optimization algorithms in GCC 4.5.0. The following code (with appropriate comments inline) demonstrate the problem. Based on a quick search, I don't think this bug has been reported yet. Note that I have assigned the component to C since I wasn't sure which other component -- rtl-optimization or tree-optimization or some other component entirely -- was the best component. I'm fairly certain that the problem is target independent since I see it with GCC 4.5.0 for mingw-w64 targets. When compiling with my system GCC (version 4.4.3) or a self-compiled GCC 4.4.4, there are no problems. Please let me know if you need any other information from me. --- #include <stdlib.h> #include <stdio.h> #include <string.h> #include <malloc.h> void *malloc_hook (size_t, const void *); void free_hook (void *, const void *); void *(* _old_malloc_hook) (size_t, const void *); void (* _old_free_hook) (void *, const void *); static int _allocCount; int main (int argc, char *argv[]) { _allocCount = 0; // Save any existing malloc hooks _old_malloc_hook = __malloc_hook; _old_free_hook = __free_hook; // Add our function as the primary __malloc_hook __malloc_hook = malloc_hook; __free_hook = free_hook; // Invoke malloc, which should invoke our function char *str = (char *)malloc(4); strcpy(str, "foo"); printf("str = %s\n", str); printf("allocCount = %d\n", _allocCount); return 0; } void *malloc_hook (size_t size, const void *caller) { void *result; // Unhook ourselves before calling malloc, // or else there will be an infinite recursion! // // *** NOTE THIS IS THE CODE THAT GETS ERRONEOUSLY // OPTIMIZED OUT RESULTING IN STACK OVERFLOW. *** __malloc_hook = _old_malloc_hook; __free_hook = _old_free_hook; // Since malloc is a function and the optimizer has // no idea what it does with __malloc_hook, it has // to ensure that the above code is generated, or // else there will be big problems. result = malloc(size); _allocCount++; // Save the current malloc hooks in case // they changed as a result of calling malloc _old_malloc_hook = __malloc_hook; _old_free_hook = __free_hook; // Add ourselves back as the hook __malloc_hook = malloc_hook; __free_hook = free_hook; return result; } void free_hook (void *ptr, const void *caller) { // Same as above but for free() __malloc_hook = _old_malloc_hook; __free_hook = _old_free_hook; free(ptr); _allocCount--; _old_malloc_hook = __malloc_hook; _old_free_hook = __free_hook; __malloc_hook = malloc_hook; __free_hook = free_hook; } --- ] x86_64-linux-gcc -g -O1 -o malloc_hook malloc_hook.c && malloc_hook Segmentation fault (core dumped) Here is the configuration options for GCC: --- Configured with: ../../gcc-4.5.0/configure --prefix=/usr/local/gcc/x86_64-linux-gcc --with-sysroot=/usr/local/gcc/x86_64-linux-gcc/x86_64-linux/sdk --program-prefix=x86_64-linux- --enable-languages=c,c++ --build=i386-linux --host=i386-linux --target=x86_64-linux --enable-multilib --enable-targets=x86_64-linux,i386-linux --disable-nls --enable-shared --without-x --- Note that the behavior also is exhibited when GCC is invoked with --sysroot=/ (basically using the system libraries instead of the sdk).
Stupid glibc extensions. __malloc_hook and __free_hook are not part of standard C90/C99. -fno-builtin-malloc will disable this optimization. Basically malloc cannot touch global memory as far as the compiler knows so it optimizes out the extra stores.
Thanks for the lightening fast response. I wouldn't have known to look there, especially since older GCC versions did not have this problem. Is it because 4.5.0 has better optimizations such that the code surrounding this malloc is now optimized away in this situation? Finally, is this issue going to be resolved within GCC, or is the permanent fix going to be to ask developers to use -fno-builtin-malloc? Thanks again.
(In reply to comment #2) > Thanks for the lightening fast response. I wouldn't have known to look there, > especially since older GCC versions did not have this problem. > > Is it because 4.5.0 has better optimizations such that the code surrounding > this malloc is now optimized away in this situation? > > Finally, is this issue going to be resolved within GCC, or is the permanent fix > going to be to ask developers to use -fno-builtin-malloc? We are going to ask developers to use -fno-builtin-malloc for now. I also think this is a glibc bug which should mark the hook variables volatile. So, can you file a bug in the glibc bugzilla as well? > Thanks again. >
(In reply to comment #3) > We are going to ask developers to use -fno-builtin-malloc for now. I also > think this is a glibc bug which should mark the hook variables volatile. I tested using the volatile keyword inside the glibc header malloc.h, and that also serves as a workaround. However, I believe the meaning of volatile is something a bit different, and using the qualifier would prevent GCC from optimizing accesses to the hook variables independent of calls to malloc. The fundamental issue seems to be that there is a mismatch between how malloc behaves and how GCC thinks it behaves. It sounds like the best way to resolve that mismatch is to use -fno-builtin-malloc, the main drawback being that the developer has to be consciously aware of this when building the code. Otherwise, there will be a nasty surprise. > So, can you file a bug in the glibc bugzilla as well? I have filed a bug there and referenced this one: http://sources.redhat.com/bugzilla/show_bug.cgi?id=11781
Moved to glibc and already fixed there.