[Bug target/39118] New: x86_64 red zone violation

["ian at airs dot com" <gcc-bugzilla at gcc dot gnu dot org>]

This is a bug report for gcc 4_3 branch.  I will attach a test case, slightly
reduced from zlib code.  When compiling this test case for the x86_64-linux
target with -O2 -fomit-frame-pointer, I see this at the start of the function:

adler32:
        pushq   %rbp
        movq    %rdi, %rax
        andl    $65535, %edi
        shrq    $16, %rax
        movq    %rsp, %rbp
        pushq   %r15
        andl    $65535, %eax
        movl    %edx, -140(%rbp)

After %rsp was copied to %rbp, %r15 was pushed.  So now %rsp is 8 bytes less
than %rbp.  The red zone is 128 bytes, so this means that any reference down to
-136(%rbp) is valid.  However, the code actually stores a value into
-140(%rbp).  This is invalid, and can cause subtle and unpredictable bugs
depending upon when the kernel interrupts this code.

This error happens because the scheduler moves this movl instruction ahead of
some other pushq instructions.  If all the pushq instructions happened first,
the stack pointer would have been decremented sufficiently that the reference
to -140(%rbp) would be safely within the red zone.

4.4 generates completely different, and seemingly better, code for this test
case.  However, I don't see anything in the 4.4 source code which would prevent
this same bug from occurring given the right test case.  So this is potentially
a serious error for the 4.4 release as well, unless it has been fixed by some
scheduling dependency that I haven't uncovered.  However, I have not been able
to find a test case.


-- 
           Summary: x86_64 red zone violation
           Product: gcc
           Version: 4.3.4
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: ian at airs dot com


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=39118