[Bug tree-optimization/81694] New: VRP optimization may introduce buffer overflow vulnerabilities into applications

["scdengyuan at gmail dot com" <gcc-bugzilla at gcc dot gnu dot org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81694

            Bug ID: 81694
           Summary: VRP optimization may introduce buffer overflow
                    vulnerabilities into applications
           Product: gcc
           Version: 7.1.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: tree-optimization
          Assignee: unassigned at gcc dot gnu.org
          Reporter: scdengyuan at gmail dot com
  Target Milestone: ---

Created attachment 41912
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=41912&action=edit
proof of concept

function extract_range_from_multiplicative_op_1 in tree-vrp.c in GNU Compiler
Collection (gcc) from 4.7.0 to 7.1.0 improperly handles integer overflow, which
might introduce buffer overflow vulnerabilities into applications.

When O2 or O3 flag is set, VRP optimization is enabled.
in my poc:
    m_index = ReadNumber(m_strm);
    if( m_index > 65535 )
        return false;

    // int m_index can be overflowed to negative but VRP think it's between
[48, +INF(OVF)], and ignored checking in next line 
    if( m_var > 0 && m_index > 0 && m_index < (1 << 16)) //this line was
optimized to "if( m_var>0 )"
    // so negative m_index can bypass the check and return true



Value ranges after VRP:

_1: ~[0B, 0B]
_2: VARYING
_3: [0, +INF]
.MEM_4: VARYING
this_7(D): ~[0B, 0B]
_9: VARYING
_11: [48, 57]  EQUIVALENCES: { _15 } (1 elements)
this_12: ~[0B, 0B]  EQUIVALENCES: { this_7(D) } (1 elements)
_15: VARYING
code.1_16: [0, +INF]
_17: [0, +INF]
val_18: [0, +INF(OVF)]
_19: [0, +INF(OVF)]
code_20: [48, 57]  EQUIVALENCES: { } (0 elements)
val_21: [48, +INF(OVF)]
_22: VARYING
code.3_23: [0, +INF]
_24: [0, +INF]
_25: [48, 57]  EQUIVALENCES: { _15 } (1 elements)
.MEM_26: VARYING
.MEM_27: VARYING
val_30: [48, +INF(OVF)]
_31: [48, 57]  EQUIVALENCES: { _22 } (1 elements)
val_32: [48, 65535]  EQUIVALENCES: { val_30 } (1 elements)

  <bb 7> [15.00%]:
  # val_30 = PHI <val_21(6)>
  # DEBUG strm => NULL
  # DEBUG val => NULL
  # DEBUG code => NULL
  this_7(D)->m_index = val_30;
  if (val_30 > 65535)
    goto <bb 10>; [46.00%]
  else
    goto <bb 8>; [54.00%]

  <bb 8> [8.10%]:
  # DEBUG strm => NULL
  # DEBUG val => NULL
  # DEBUG code => NULL
  _2 = this_7(D)->m_var;
  if (_2 > 0)
    goto <bb 9>; [67.61%]
  else
    goto <bb 10>; [32.39%]

  <bb 9> [3.70%]:
  # DEBUG result => 1