[Bug sanitizer/83356] [7 Regression] excessive stack usage compiling with -O2 -fsanitize=bounds -fsanitize=object-size

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=83356

--- Comment #9 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
Created attachment 42921
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=42921&action=edit
gcc7-pr83356.patch

Untested patch for the second sanopt.  This doesn't change anything on this
testcase though, because we don't sanopt_optimize IFN_UBSAN_BOUNDS nor
IFN_UBSAN_OBJECT_SIZE.  The latter is sometimes (in 230 out of 702 cases on
this testcase) folded away during objsz2 pass and can't be folded or redundancy
optimized earlier anyway.  On this testcase, I don't think most of the other
calls are actually redundant, what is true is that most of the UBSAN_BOUNDS
calls use highest array index 255 and the indexes are unsigned_int >> 24 or
some other extractions of unsigned char.  By using value ranges we could get
rid of most of them, but I'm actually hesistant here, because value ranges are
computed with the assumption UB doesn't happen in the program, while sanitizers
are the way to detect those UBs.  Unfortunately, we don't have different kinds
of ranges, ranges that are proven for any kind of execution, including ones
with UB, vs. ranges that derive from lack of UB.  In cases like:
  _74 = _1197 & 255;
  _419 = (int) _74;
  UBSAN_BOUNDS (0B, _419, 255);
or:
  _360 = _1197 >> 24;
  _428 = (int) _360;
  UBSAN_BOUNDS (0B, _428, 255);
or:
  _1200 = (unsigned char) _1199;
  _372 = (int) _1199;
  UBSAN_BOUNDS (0B, _372, 255);
perhaps we could at some point remove those ifns, because no matter what the
arguments will provably be all >= 0 and <= 255.  Though, without using full VR
that sounds quite hackish and in many other real-world cases the array size
won't be that nice and we won't be able to prove anything.