[Bug c/67999] New: Wrong optimization of pointer comparisons

["ch3root at openwall dot com" <gcc-bugzilla at gcc dot gnu dot org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=67999

            Bug ID: 67999
           Summary: Wrong optimization of pointer comparisons
           Product: gcc
           Version: 5.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c
          Assignee: unassigned at gcc dot gnu.org
          Reporter: ch3root at openwall dot com
  Target Milestone: ---

The following program:

#include <stdio.h>
#include <stdlib.h>

int main(void)
{
  size_t len = 0xC0000000; /* 3GB */
  char *buf = malloc(len);
  if (!buf)
    return 1;

  printf("buf       = %p\n", (void *)buf);
  printf("len       = 0x%08zx\n", len);
  printf("buf + len = %p\n", (void *)(buf + len));

  printf("buf + len < buf: ");
  if (buf + len < buf)
    printf("true\n");
  else
    printf("false\n");

  return 0;
}

generates output like this:

$ gcc -m32 -O2 test.c && ./a.out
buf       = 0x3751d008
len       = 0xc0000000
buf + len = 0xf751d008
buf + len < buf: true

The result of the comparison "buf + len < buf" is wrong.

AFAICT the comparison "buf + len < buf" is simplified into "len < 0" where
"len" is treated as a signed value which is wrong when "len" is of an unsigned
type and has a large value.