This is the mail archive of the mailing list for the GCC project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Signed int overflow behaviour in the security context

Andreas Bogk <> writes:

> The case is clear when an if branch is folded away, because some
> comparison is folded that would not have been under twos-complement
> semantics.  This is a pattern that probably doesn't happen all that much
> in code out there, except in post-facto overflow checking.

Unfortunately this happens quite a lot.

Consider code along these lines:

struct s { int len; char* p; };

inline char
bar (struct s *sp, int n)
  if (n < 0)
    abort ();
  if (n > sp->len)
    abort ();
  return sp->p[n];

foo (struct s *sp, int n)
  int len = sp->len;
  int i;
  int tot = 0;
  for (i = 0; i <= len; ++i)
    tot += bar (sp, i);
  return tot;

Let's assume that bar() is inlined into foo().  Now consider the
assert.  If signed overflow is undefined, then we can optimize away
the "n < 0" test; it will never be true.  If signed overflow is
defined, then we can not optimize that away.  That is because as far
as the compiler knows, sp->len might be INT_MAX.  In that case, the
loop will never terminate, and i will wrap and become negative.  (The
compiler may also eliminate the "n > sp->len" test, but that does not
rely on undefined signed overflow.)

This is a typical example of removing an if branch because signed
overflow is undefined.  This kind of code is common enough.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]