[Bug sanitizer/70051] New: ubsan doesn't detect VLA overflow

[msebor at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=70051

            Bug ID: 70051
           Summary: ubsan doesn't detect VLA overflow
           Product: gcc
           Version: 6.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: sanitizer
          Assignee: unassigned at gcc dot gnu.org
          Reporter: msebor at gcc dot gnu.org
                CC: dodji at gcc dot gnu.org, dvyukov at gcc dot gnu.org,
                    jakub at gcc dot gnu.org, kcc at gcc dot gnu.org
  Target Milestone: ---

The undefined behavior sanitizer tries to detect some simple invalid uses of
VLAs (negative bounds, AFAICS) but misses overflow/wraparound in the unsigned
bounds.  For example, the test case below crashes (or aborts when the memset
loop is removed), when one would expect or hope to have the sanitizer
instrumentation to detect this.

$ cat z.c && /home/msebor/build/gcc-trunk-svn/gcc/xg++
-B/home/msebor/build/gcc-trunk-svn/gcc  -L
/home/msebor/build/gcc-trunk-svn/stage3-x86_64-pc-linux-gnu/libstdc++-v3/src/.libs
-L
/home/msebor/build/gcc-trunk-svn/stage3-x86_64-pc-linux-gnu/libsanitizer/ubsan/.libs
-O2 -Wall -Wextra -Wpedantic -fsanitize=undefined -xc++ z.c &&
LD_LIBRARY_PATH=/home/msebor/build/gcc-trunk-svn/stage3-x86_64-pc-linux-gnu/libsanitizer/ubsan/.libs
./a.out 
typedef __SIZE_TYPE__ size_t;

void __attribute__ ((noclone, noinline))
foo (void *p) { }

void __attribute__ ((noclone, noinline))
bar (size_t m, size_t n)
{
    int a [m][n];
    for (size_t i = 0; i != m; ++i)
        __builtin_memset (a [i], 0, n * sizeof (int));
    foo (a);
}

#define M (__SIZE_MAX__ / 1024)
#define N (__SIZE_MAX__ / 1024)

int main (void)
{
#if __cplusplus
    try {
        bar (M, N);
        __builtin_abort ();
    }
    catch (...) {
    }
#else
    bar (M, N);
#endif
}

z.c: In function ‘void foo(void*)’:
z.c:4:12: warning: unused parameter ‘p’ [-Wunused-parameter]
 foo (void *p) { }
            ^
z.c: In function ‘void bar(size_t, size_t)’:
z.c:9:16: warning: ISO C++ forbids variable length array ‘a’ [-Wvla]
     int a [m][n];
                ^
z.c:9:16: warning: ISO C++ forbids variable length array ‘a’ [-Wvla]
Bus error (core dumped)