[Bug target/77333] New: Incorrect stack adjust in epilogue when targeting i686-w64-mingw32

["kfischer at college dot harvard.edu" <gcc-bugzilla at gcc dot gnu dot org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77333

            Bug ID: 77333
           Summary: Incorrect stack adjust in epilogue when targeting
                    i686-w64-mingw32
           Product: gcc
           Version: 6.1.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: kfischer at college dot harvard.edu
  Target Milestone: ---

This is reduced from https://github.com/JuliaLang/julia/issues/18162.
It appears that GCC miscompiles LLVM when targeting i686-w64-mingw32.
This behavior was observed on at least GCC 5.4.0 and 6.1.0.
GCC 4.9 was confirmed to not have this problem.

To reproduce:
```
git clone https://github.com/llvm-mirror/llvm
(cd llvm && git checkout release_37)
mkdir llvm-build
(cd llvm-build && \
../llvm/configure --build=x86_64-linux-gnu --host=i686-w64-mingw32
--enable-optimized CC=i686-w64-mingw32-gcc CXX="i686-w64-mingw32-g++ -g1
-funwind-tables -fasynchronous-unwind-tables" --enable-shared && \
make -j8)
cp llvm-build/Release+Asserts/bin/opt.exe
llvm-build/Release+Asserts/bin/LLVM-3.7.dll .
cp ~/cross-w32/i686-w64-mingw32/lib/libgcc_s_sjlj-1.dll
~/cross-w32/i686-w64-mingw32/lib/libstdc++-6.dll  .
wine ./opt.exe -slp-vectorizer -S
~/llvm/test/Transforms/SLPVectorizer/X86/vector.ll
```
(-g and unwind tables are only to make debugging easier, also happens without)

The problem appears to be the following:
```
Stopped in function
__ZN12_GLOBAL__N_113SLPVectorizer18tryToVectorizePairEPN4llvm5ValueES3_RNS_7BoUpSLPE.part.774
   0x00000000025cf853<+51>:     leal    24(%esp), %eax
   0x00000000025cf857<+55>:     calll   -2284
=> 0x00000000025cf85c<+60>:     subl    $12, %esp
   0x00000000025cf85f<+63>:     addl    $44, %esp
   0x00000000025cf862<+66>:     retl    $12
```
This code jumps to an invalid address once it hits the `retl`.
I believe the `subl 12` is incorrect, perhaps supposed to clean up a
corresponding retl $12` in the callee, when there actually isn't one [^1].
Moreover, stepping back one instruction (to the end of function being called
here) shows the stack intact and a plausible backtrace, lending further
credibility to this theory:
```
1|debug > rsi
Stopped in function
__ZN12_GLOBAL__N_113SLPVectorizer18tryToVectorizeListEN4llvm8ArrayRefIPNS1_5ValueEEERNS_7BoUpSLPES5_b.constprop.793
   0x00000000025cf051<+225>:    popl    %edi
   0x00000000025cf052<+226>:    popl    %ebp
=> 0x00000000025cf053<+227>:    retl
   0x00000000025cf054<+228>:    addl    $4, %ebx
   0x00000000025cf057<+231>:    cmpl    %ebx, %esi
1|debug > bt
[1]
__ZN12_GLOBAL__N_113SLPVectorizer18tryToVectorizeListEN4llvm8ArrayRefIPNS1_5ValueEEERNS_7BoUpSLPES5_b.constprop.793
[2]
__ZN12_GLOBAL__N_113SLPVectorizer18tryToVectorizePairEPN4llvm5ValueES3_RNS_7BoUpSLPE.part.774
[3]
__ZN12_GLOBAL__N_113SLPVectorizer22vectorizeChainsInBlockEPN4llvm10BasicBlockERNS_7BoUpSLPE
[4] __ZN12_GLOBAL__N_113SLPVectorizer13runOnFunctionERN4llvm8FunctionE.part.786
[5] __ZN4llvm13FPPassManager13runOnFunctionERNS_8FunctionE
[6] __ZN4llvm13FPPassManager11runOnModuleERNS_6ModuleE
[7] __ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE
[8] Most likely <Unknown Module>
```
(full assembly of that function is here:
https://gist.github.com/Keno/cb45c3f1e925c6ae3484ca98f7300a4a)

Lastly, looking at the CFI of the crashing frame shows that it indeed expected
the stack pointer to have been adjusted up by 12 bytes after the call:
```
Stopped in function
__ZN12_GLOBAL__N_113SLPVectorizer18tryToVectorizePairEPN4llvm5ValueES3_RNS_7BoUpSLPE.part.774
   0x00000000025cf853<+51>:     leal    24(%esp), %eax
   0x00000000025cf857<+55>:     calll   -2284
=> 0x00000000025cf85c<+60>:     subl    $12, %esp
   0x00000000025cf85f<+63>:     addl    $44, %esp
   0x00000000025cf862<+66>:     retl    $12
1|debug > cfi
DW_CFA_def_cfa %esp 4
DW_CFA_offset %eip 1
DW_CFA_nop
DW_CFA_nop
--------------

DW_CFA_advance_loc 3 (=> 3)
DW_CFA_def_cfa_offset 48
DW_CFA_advance_loc 57 (=> 60)
DW_CFA_def_cfa_offset 36
DW_CFA_advance_loc 3 (=> 63)
--------------
DW_CFA_def_cfa_offset 48
DW_CFA_advance_loc 3 (=> 66)
DW_CFA_def_cfa_offset 4
```

[^1]: such cleanup happens when calling
__ZN12_GLOBAL__N_113SLPVectorizer18tryToVectorizePairEPN4llvm5ValueES3_RNS_7BoUpSLPE.part.774
```
1|debug > f 3
Stopped in function
__ZN12_GLOBAL__N_113SLPVectorizer22vectorizeChainsInBlockEPN4llvm10BasicBlockERNS_7BoUpSLPE
   0x00000000025d0727<+3319>:   movl    %ecx, 8(%esp)
   0x00000000025d072b<+3323>:   movl    40(%esp), %ecx
=> 0x00000000025d072f<+3327>:   calll   -3860
   0x00000000025d0734<+3332>:   subl    $12, %esp
   0x00000000025d0737<+3335>:   testb   %al, %al
```

Please let me know if I can provide any further information.