[Bug target/16601] New: [3.3 regression] miscompilation of automatic dynamic arrays in crypto/IPsec subsystems of the Linux kernel

["debian-gcc at lists dot debian dot org" <gcc-bugzilla at gcc dot gnu dot org>]

[forwarded from http://bugs.debian.org/259887]

regression from 3.2.3, rechecked with 3.3.4, fixed in 3.4.

    Matthias

Herbert Xu writes:

With the options -O2 -mpreferred-stack-boundary=2 (default flags for kernel
compilation, gcc 3.3.4 is miscompiling automatic dynamic arrays.  Unfortunately
both are used in the crypto/IPsec subsystems of the Linux kernel. 
 
Here is a sample program: 

#include <string.h> 
 
int bar(char *s); 
 
int foo(char *s, int len, int x) 
{ 
        char buf[x ? len : 0]; 
 
        if (x) { 
                memcpy(buf, s, len); 
                s = buf; 
        } 
 
        return bar(s); 
} 
 
With gcc 3.3.4, this produces: 

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 
        .file   "b.c" 
        .text 
        .p2align 4,,15 
.globl foo 
        .type   foo, @function 
foo: 
        pushl   %ebp 
        xorl    %eax, %eax 
        movl    %esp, %ebp 
        subl    $24, %esp 
        movl    16(%ebp), %ecx 
        movl    %edi, -4(%ebp) 
        movl    12(%ebp), %edx 
        movl    %esp, %edi 
        movl    %ebx, -12(%ebp) 
        movl    %esi, -8(%ebp) 
        decl    %edx 
        movl    8(%ebp), %esi 
        testl   %ecx, %ecx 
        setne   %al 
        decl    %eax 
        orl     %eax, %edx 
        addl    $19, %edx 
        andl    $-4, %edx 
--------------------------------------------------------------------- 
        subl    %edx, %esp 
        leal    27(%esp), %ebx 
        andl    $-16, %ebx 
 
Note the offset 27.  The same program when compiled with gcc 3.2.3 
produces similar output but it uses an offset of 15. 
 
Suppose that len = 16, x != 0, and %esp & 15 = 8 before the subl. 
 
That means %edx = (15 + 19) & ~3 = 32.  So %esp & 15 is still 8 
after the subtraction.  That is, %esp = 16x + 8.  Hence 
%ebx = (%esp + 27) & ~15 = (16x + 35) & ~15 = 16x + 32 = %esp + 24. 
 
Therefore buf will only contain 8 bytes of space instead of 16 
bytes. 
--------------------------------------------------------------------- 
        testl   %ecx, %ecx 
        jne     .L5 
.L4: 
        movl    %esi, (%esp) 
        call    bar 
        movl    %edi, %esp 
        movl    -12(%ebp), %ebx 
        movl    -8(%ebp), %esi 
        movl    -4(%ebp), %edi 
        movl    %ebp, %esp 
        popl    %ebp 
        ret 
        .p2align 4,,7 
.L5: 
        movl    12(%ebp), %eax 
        movl    %esi, 4(%esp) 
        movl    %ebx, %esi 
        movl    %eax, 8(%esp) 
        movl    %ebx, (%esp) 
        call    memcpy 
        jmp     .L4 
        .size   foo, .-foo 
        .section        .note.GNU-stack,"",@progbits 
        .ident  "GCC: (GNU) 3.3.4 (Debian 1:3.3.4-3)" 
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 
 
Since this bug can lead to remotely triggered crashes and possibly 
exploits I'm rating it as critical.

-- 
           Summary: [3.3 regression] miscompilation of automatic dynamic
                    arrays in crypto/IPsec subsystems of the Linux kernel
           Product: gcc
           Version: 3.3.4
            Status: UNCONFIRMED
          Severity: critical
          Priority: P2
         Component: target
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: debian-gcc at lists dot debian dot org
                CC: gcc-bugs at gcc dot gnu dot org,gdr at gcc dot gnu dot
                    org,herbert at gondor dot apana dot org dot au
 GCC build triplet: ix86-linux
  GCC host triplet: ix86-linux
GCC target triplet: ix86-linux


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=16601