[Bug sanitizer/65435] New: UBsan runtime error reports in OpenSSL aes_core.c

[bernd.edlinger at hotmail dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=65435

            Bug ID: 65435
           Summary: UBsan runtime error reports in OpenSSL aes_core.c
           Product: gcc
           Version: 5.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: sanitizer
          Assignee: unassigned at gcc dot gnu.org
          Reporter: bernd.edlinger at hotmail dot de
                CC: dodji at gcc dot gnu.org, dvyukov at gcc dot gnu.org,
                    jakub at gcc dot gnu.org, kcc at gcc dot gnu.org

Hi,

I am not quite sure if this is a bug in OpenSSL or in UBSan.
This gets reported by ubsan in OpenSSL 1.0.0m 5 Jun 2014:

aes_core.c:1144:30: runtime error: left shift of 136 by 24 places cannot be
represented in type 'int'
aes_core.c:1151:30: runtime error: left shift of 158 by 24 places cannot be
represented in type 'int'
aes_core.c:1137:30: runtime error: left shift of 239 by 24 places cannot be
represented in type 'int'
aes_core.c:1130:30: runtime error: left shift of 139 by 24 places cannot be
represented in type 'int'


when I look at that lines, I see the following (repeated 4 times):

        s0 =
                (Td4[(t0 >> 24)       ] << 24) ^
                (Td4[(t3 >> 16) & 0xff] << 16) ^
                (Td4[(t2 >>  8) & 0xff] <<  8) ^
                (Td4[(t1      ) & 0xff])       ^
                rk[0];

and
static const u8 Td4[256] = {
    0x52U, 0x09U, 0x6aU, 0xd5U, 0x30U, 0x36U, 0xa5U, 0x38U, ...

I assume u8 means unsigned char.
So are we correct to convert u8 to int before << 24,
or should it be u8 to unsigned int before << 24, what OpenSSL
apparently expects?