GCC Bugzilla Bug 37750

while testing ffmpeg compiled with gcc 4.3.3 from 4.3 trunk
I got a lot of crashes especially in snow encoding testcase

by using:
-fno-tree-dominator-opts -fno-tree-vrp -fno-dce -fno-tree-ch

I get no crashes

gcc 4.4 has same problems,

I've already filled a bug about tree-ch, that corrupts the stack
on win32 (looks like it's related to alloca())

No other target has this issue except Win32.  I am going to say mingw support
for alloca is broken somehow.

this problem started with 4.3, 4.3.2 on debian linux hasn't these problems 

As usual we would need preprocessed source as a testcase.

unfortunately snow.c is a very big file, I'll try to find a shorter example,
but if the problem is in alloca() the generated asm will not be so usefull

*** Bug 37584 has been marked as a duplicate of this bug. ***

first bt, (pls tell me if you need output of leave temps, generated asm
preprocessed or other stuff)

Program received signal SIGSEGV, Segmentation fault.
0x0092c4c2 in _alloca ()
(gdb) bt
#0  0x0092c4c2 in _alloca ()
#1  0x005c7462 in get_dc (s=0x6230030, mb_x=0, mb_y=0, plane_index=0) at
libavcodec/snow.c:2426
#2  0x005cff6a in iterative_me (s=0x6230030) at libavcodec/snow.c:3126
#3  0x005d7467 in encode_blocks (s=0x6230030, search=1) at
libavcodec/snow.c:2074
#4  0x005d7a2a in encode_frame (avctx=0x5aa3f90,
    buf=0x5fe06c0 "<BINARY DATA REMOVED>"..., buf_size=262144, data=0x22eda8)
at libavcodec/snow.c:4268
#5  0x00499513 in avcodec_encode_video (avctx=0x5aa3f90,
    buf=0x5fe06c0 "<BINARY DATA REMOVED>"..., buf_size=262144, pict=0x22eda8)
at libavcodec/utils.c:894
#6  0x00405626 in output_packet (ist=0x5ed6450, ist_index=0,
ost_table=0x5aa4390, nb_ostreams=1, pkt=0x22fed8) at ffmpeg.c:954
#7  0x00409269 in av_encode (output_files=0xb6b378, nb_output_files=1,
input_files=0xb6a608, nb_input_files=1, stream_maps=0xb6b3c8, nb_stream_maps=0)
    at ffmpeg.c:2116
#8  0x00409761 in main (argc=Cannot access memory at address 0x0
) at ffmpeg.c:3888

compile flags

OPTFLAGS=-O3 -fno-common -g3 -march=native -mtune=native -pipe
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_ISOC99_SOURCE
-D_POSIX_C_SOURCE=200112 -fasm -std=c99 -fomit-frame-pointer -DPTW32_STATIC_LIB
-g3 -Wdeclaration-after-statement -Wall -Wno-switch -Wdisabled-optimization
-Wpointer-arith -Wredundant-decls -Wno-pointer-sign -Wcast-qual -Wwrite-strings
-Wtype-limits -O2 -fno-math-errno -fno-signed-zeros

-O3 -fno-common -march=native -mtune=native -pipe where added by me
(-fno-common is needed to avoid crash in sse code)

another crash in snow, this time argc/argv is not screwed

Program received signal SIGSEGV, Segmentation fault.
0x0092c4c2 in _alloca ()
(gdb) bt
#0  0x0092c4c2 in _alloca ()
#1  0x005d7de7 in encode_frame (avctx=0x5aa3f20,
    buf=0x5fda480 "<BINARY DATA REMOVED>"..., buf_size=405504, data=0x22eda8)
at libavcodec/snow.c:2426
#2  0x00499513 in avcodec_encode_video (avctx=0x5aa3f20,
    buf=0x5fda480 "<BINARY DATA REMOVED>"..., buf_size=405504, pict=0x22eda8)
at libavcodec/utils.c:894
#3  0x00405626 in output_packet (ist=0x5ed6450, ist_index=0,
ost_table=0x5cd6470, nb_ostreams=1, pkt=0x22fed8) at ffmpeg.c:954
#4  0x00409269 in av_encode (output_files=0xb6b378, nb_output_files=1,
input_files=0xb6a608, nb_input_files=1, stream_maps=0xb6b3c8, nb_stream_maps=0)
    at ffmpeg.c:2116
#5  0x00409761 in main (argc=95043360, argv=0x3f4998) at ffmpeg.c:3888

sqv1

Program received signal SIGSEGV, Segmentation fault.
0x0092c4c2 in _alloca ()
(gdb) bt
#0  0x0092c4c2 in _alloca ()
#1  0x005e3338 in svq1_encode_plane (s=0x5a9f2c0, plane=<value optimized out>,
    src_plane=0x5ac9fa0 "<BINARY DATA REMOVED>"..., ref_plane=0x6063010 '\200'
<repeats 200 times>...,
    decoded_plane=0x6089300 '\200' <repeats 200 times>..., width=352,
height=288, src_stride=352, stride=416) at libavcodec/svq1enc.c:363
#2  0x005e467a in svq1_encode_frame (avctx=0x5aa3eb0, buf=0x5fe05a0 "",
buf_size=405504, data=0x22eda8) at libavcodec/svq1enc.c:541
#3  0x00499513 in avcodec_encode_video (avctx=0x5aa3eb0, buf=0x5fe05a0 "",
buf_size=405504, pict=0x22eda8) at libavcodec/utils.c:894
#4  0x00405626 in output_packet (ist=0x5ed6450, ist_index=0,
ost_table=0x5ac9e00, nb_ostreams=1, pkt=0x22fed8) at ffmpeg.c:954
#5  0x00409269 in av_encode (output_files=0xb6b378, nb_output_files=1,
input_files=0xb6a608, nb_input_files=1, stream_maps=0xb6b3c8, nb_stream_maps=0)
    at ffmpeg.c:2116
#6  0x00409761 in main (argc=Cannot access memory at address 0x1000000
) at ffmpeg.c:3888

Does this issue appears also, when using builtin alloca version? As I noticed
does the switch -fno-builtin shows explict broken _alloca for x64. The
call-save area isn't adjusted and compiler seems not to take care here

I'm using 4.5 from svn, with -O2 and looks like not affected
4.3 and 4.4 are almost unusable on mingw (at least my builds)
something changed in 4.5 branch,
I've not tested further 4.3 or .4.4 since I was using 4.2 then switched to 4.5
when it started to work on mingw

however 4.5 is still far from being stable as 4.2, I get many crashes while
using complied mplayer (it's a stress test for gcc :))

Unfortunately I had no much time to debug mplayer builds