[Bug rtl-optimization/79149] New: bad optimization on MIPS and ARM leading to excessive stack usage in some cases

[arnd at linaro dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=79149

            Bug ID: 79149
           Summary: bad optimization on MIPS and ARM leading to excessive
                    stack usage in some cases
           Product: gcc
           Version: 4.9.3
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: rtl-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: arnd at linaro dot org
  Target Milestone: ---

Created attachment 40546
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=40546&action=edit
compressed preprocessed source for wp512 crypto from linux kernel

Build-testing the Linux kernel on MIPS shows warnings about possible kernel
stack overflow in the kernel's crypto libraries, in particular the wp512
algorithm, as the stack frame grows beyond the normal per-function limit of
1024 bytes we use in the kernel.

I have been able to avoid the problem by turning off two specific
optimizations,
and I have reproduced the same thing with both gcc-4.9 and gcc-7.0:

$ /home/arnd/cross-gcc/bin/mips-linux-gcc-7.0.0 -fno-strict-aliasing -O2 -c
-Wall -Wframe-larger-than=100 -Wno-pointer-sign  wp512.i  
../../crypto/wp512.c: In function 'wp512_process_buffer':
../../crypto/wp512.c:987:1: warning: the frame size of 1128 bytes is larger
than 100 bytes [-Wframe-larger-than=]

$ /home/arnd/cross-gcc/bin/mips-linux-gcc-7.0.0 -fno-strict-aliasing -O2 -c
-Wall -Wframe-larger-than=100 -Wno-pointer-sign  wp512.i 
-fno-sched-critical-path-heuristic -fno-sched-dep-count-heuristic
../../crypto/wp512.c: In function 'wp512_process_buffer':
../../crypto/wp512.c:987:1: warning: the frame size of 304 bytes is larger than
100 bytes [-Wframe-larger-than=]

$ /home/arnd/cross-gcc/bin/mips-linux-gcc-4.9.3 -fno-strict-aliasing -O2 -c
-Wall -Wframe-larger-than=100 -Wno-pointer-sign  wp512.i  
../../crypto/wp512.c: In function 'wp512_process_buffer':
../../crypto/wp512.c:987:1: warning: the frame size of 1096 bytes is larger
than 100 bytes [-Wframe-larger-than=]

$ /home/arnd/cross-gcc/bin/mips-linux-gcc-4.9.3 -fno-strict-aliasing -O2 -c
-Wall -Wframe-larger-than=100 -Wno-pointer-sign  wp512.i 
-fno-sched-critical-path-heuristic -fno-sched-dep-count-heuristic 
../../crypto/wp512.c: In function 'wp512_process_buffer':
../../crypto/wp512.c:987:1: warning: the frame size of 272 bytes is larger than
100 bytes [-Wframe-larger-than=]

To cross-check the problem, I have tried compiling the same file on ARM, which
shows similar results but stays below the warning limit:

/home/arnd/cross-gcc/bin/arm-linux-gnueabi-gcc-7.0.0 -fno-strict-aliasing -O2
-c -Wall -Wframe-larger-than=100 -Wno-pointer-sign  wp512.i  
../../crypto/wp512.c: In function 'wp512_process_buffer':
../../crypto/wp512.c:987:1: warning: the frame size of 816 bytes is larger than
100 bytes [-Wframe-larger-than=]

$ /home/arnd/cross-gcc/bin/arm-linux-gnueabi-gcc-7.0.0 -fno-strict-aliasing -O2
-c -Wall -Wframe-larger-than=100 -Wno-pointer-sign  wp512.i 
-fno-sched-critical-path-heuristic -fno-sched-dep-count-heuristic
../../crypto/wp512.c: In function 'wp512_process_buffer':
../../crypto/wp512.c:987:1: warning: the frame size of 344 bytes is larger than
100 bytes [-Wframe-larger-than=]

$ /home/arnd/cross-gcc/bin/arm-linux-gnueabi-gcc-4.9.3 -fno-strict-aliasing -O2
-c -Wall -Wframe-larger-than=100 -Wno-pointer-sign  wp512.i  
../../crypto/wp512.c: In function 'wp512_process_buffer':
../../crypto/wp512.c:987:1: warning: the frame size of 840 bytes is larger than
100 bytes [-Wframe-larger-than=]

$ /home/arnd/cross-gcc/bin/arm-linux-gnueabi-gcc-4.9.3 -fno-strict-aliasing -O2
-c -Wall -Wframe-larger-than=100 -Wno-pointer-sign  wp512.i  
-fno-sched-critical-path-heuristic -fno-sched-dep-count-heuristic
../../crypto/wp512.c: In function 'wp512_process_buffer':
../../crypto/wp512.c:987:1: warning: the frame size of 376 bytes is larger than
100 bytes [-Wframe-larger-than=]

However, using an x86 compiler, the frame for the same source is always under
300 bytes, and the options have no effect.