[Bug c/68065] New: Size calculations for VLAs can overflow

[ch3root at openwall dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=68065

            Bug ID: 68065
           Summary: Size calculations for VLAs can overflow
           Product: gcc
           Version: 5.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c
          Assignee: unassigned at gcc dot gnu.org
          Reporter: ch3root at openwall dot com
  Target Milestone: ---

The following program crashes while writing to a buffer:

#include <stdint.h>
#include <stdio.h>

int main(void)
{
  size_t size = SIZE_MAX / sizeof(int) + 2;
  int buf[size];

  printf("%zu\n", sizeof(buf));
  for (size_t i = 0; i < size; i++)
    buf[i] = 1;

  return 0;
}

(Compile without optimization or make sure the loop is not optimized away.)

It would be better to detect an overflow in the size calculation and crash
right away, before any harm is done.

While at it, size of VLAs could probably be limited to PRTDIFF_MAX to be in
line with ordinary arrays.