[Bug middle-end/66876] New: vrp: changing unsigned to signed comparison

[konstantin.vladimirov at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=66876

            Bug ID: 66876
           Summary: vrp: changing unsigned to signed comparison
           Product: gcc
           Version: 5.1.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: middle-end
          Assignee: unassigned at gcc dot gnu.org
          Reporter: konstantin.vladimirov at gmail dot com
  Target Milestone: ---

Minimized test case (note explicit unsigned long cast):

---
#include <stdlib.h>

int a = 0xecfb39f5;
unsigned short *d = (unsigned short *) &a;

int
main (void)
{
  if ((unsigned long) (65536 * d[1] + d[0]) < (1UL << 28))
    {
      abort ();
    }

  return 0;
}
---

Compiler version:

---
$ ~/x86-toolchain-5.1/bin/gcc -v
Using built-in specs.
COLLECT_GCC=/home/tilir/x86-toolchain-5.1/bin/gcc
COLLECT_LTO_WRAPPER=/home/tilir/x86-toolchain-5.1/libexec/gcc/x86_64-unknown-linux-gnu/5.1.0/lto-wrapper
Target: x86_64-unknown-linux-gnu
Configured with: ../gcc-5.1-src/gcc_5_1_0_release/configure
--enable-languages=c,c++ --prefix=/home/tilir/x86-toolchain-5.1/
Thread model: posix
gcc version 5.1.0 (GCC) 
---

$ ~/x86-toolchain-5.1/bin/gcc -O2 test.c
$ ./a.out 
Aborted

but

$ ~/x86-toolchain-5.1/bin/gcc -O2 test.c -fno-tree-vrp
$ ./a.out

Everything is ok.

Before VRP:

  _9 = (unsigned int) _8;
  if (_9 <= 268435455)
    goto <bb 3>;
  else
    goto <bb 4>;


On VRP stage:

_8 : --> single use.
_9 = (unsigned int) _8;

_9 : --> single use.
if (_9 <= 268435455)

After VRP:

  if (_8 <= 268435455)
    goto <bb 3>;
  else
    goto <bb 4>;

By C11 standard, the rank of unsigned long is equal to rank of long and higher,
then rank of int, so integer promotions here should make unsigned long from
both sides.