[Bug c/38001] New: regression in 4.3: alignment checks wrongly optimized away (runtime failure)

[thomas dot orgis at awi dot de]

For mpg123 I use __attribute__((aligned(16))) for variables that are possibly
used by SSE code. Now, I also create a library containing that SSE code, which
can be used from third party applications.
The use case of linking gcc-compiled mpg123 code with another compiler that did
_not_ align the stack at 16 bytes lead to the implementation of a runtime
alignment check in the toplevel API calls of the library.

I stripped this down to the example I am about to attach.
Please note that I observed the issue in my library code first, so it should
not be bound to problems with the stack in main() -- though the example has the
code in main().

Basic code fragment:

        double __attribute__((aligned(16))) altest[1];
        double __attribute__((aligned(16))) altest2[1];
        if((size_t)altest % 16 != 0 || (size_t)altest2 % 16 != 0)
        printf("not aligned with %!\n");

If the stack from the caller is not proper, I expected this check to fire.
Now a specific user who had -mpreferred-stack-boundary=2 in his CFLAGS pointed
me to reinvestigate the matter:

Since that option apparently overrides the __attribute__((aligned(16))) (a bug
or misfeature on its own, IMHO), the user encountered a segfault on using the
misaligned memory in SSE code.
That made me realize that I not just have to check for misaligned stack from
the caller but also for the alignment working at all in one stack frame (hence,
the two variables altest and altest2). Now, I updated the check and noticed
that gcc-4.3.1 -mpreferred-stack-boundary=2 still produced a crashing program
instead of triggering the check at runtime.

I compared with gcc-3.3.6 and gcc-3.4.6; both versions correctly detected the
bad alignment at runtime and stopped in a controlled way. This led to
investigations with the stripped down code.

Results:
1. gcc-4.3.1 optimizes away the altest % 16 or altest & 15 to plain zero, under
the wrong assumption that the alignment to 16 bytes works.
2. consequently, the if(altest % 16 != 0 ...) check is totally optimized away
-- even with -O0!
2. additonally, the & 15 check is miscompiled by gcc-3.4.6: it boils down to
  and $0x1,%eax
instead of
  and $0xf,%eax
...please see the attached files, with disassembly. This is inconsistent,
even...

Actually, I am not sure how many bugs I am raising here. The main point is:
GCC should not omptimize away a computation which has an unknown result at
compile time -- or at least take into account that it actually does so against
it's own actions that render the predicted result invalid (align the stack at a
different boundary).

Footnote: I know that it is a bad idea to have -mpreferred-stack-boundary=2 in
this case, but this shows that the check is miscompiled and that insight also
applies to the correctly compiled library being linked with other code that
aligns stack differently.


-- 
           Summary: regression in 4.3: alignment checks wrongly optimized
                    away (runtime failure)
           Product: gcc
           Version: 4.3.1
            Status: UNCONFIRMED
          Severity: major
          Priority: P3
         Component: c
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: thomas dot orgis at awi dot de
 GCC build triplet: i686-pc-linux-gnu
  GCC host triplet: i686-pc-linux-gnu
GCC target triplet: i686-pc-linux-gnu


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=38001