Re: Signed int overflow behaviour in the security context

[Ian Lance Taylor <iant at google dot com>]

Andreas Bogk <andreas@andreas.org> writes:

> The case is clear when an if branch is folded away, because some
> comparison is folded that would not have been under twos-complement
> semantics.  This is a pattern that probably doesn't happen all that much
> in code out there, except in post-facto overflow checking.

Unfortunately this happens quite a lot.

Consider code along these lines:

struct s { int len; char* p; };

inline char
bar (struct s *sp, int n)
{
  if (n < 0)
    abort ();
  if (n > sp->len)
    abort ();
  return sp->p[n];
}

void
foo (struct s *sp, int n)
{
  int len = sp->len;
  int i;
  int tot = 0;
  for (i = 0; i <= len; ++i)
    tot += bar (sp, i);
  return tot;
}

Let's assume that bar() is inlined into foo().  Now consider the
assert.  If signed overflow is undefined, then we can optimize away
the "n < 0" test; it will never be true.  If signed overflow is
defined, then we can not optimize that away.  That is because as far
as the compiler knows, sp->len might be INT_MAX.  In that case, the
loop will never terminate, and i will wrap and become negative.  (The
compiler may also eliminate the "n > sp->len" test, but that does not
rely on undefined signed overflow.)

This is a typical example of removing an if branch because signed
overflow is undefined.  This kind of code is common enough.

Ian