Signed int overflow behaviour in the security context

[Andreas Bogk <andreas at andreas dot org>]

Dear compiler hackers,

I feel compelled to revisit a topic that has already been extensively
discussed on this mailing list, unfortunately with an outcome that is
highly dangerous to the security of about any existing system compiled
with gcc in the world.

It is my fear that the existing behaviour of gcc when used without
-fwrapv breaks a lot of code out there that was written with the
implicit assumption that signed ints would overflow the way the
underlying machine integers do.  More importantly, some of the code that
breaks is in checks against integer overflows, and thus relevant to
security: breaking the checks means opening vulnerabilities.  A number
of IT security professionals shares my view of things here.

I am well aware that the current behaviour of gcc is within what is
allowed by the C standard.  I am also aware that writing code depending
on a certain type of signed overflow behaviour is not.  However, there
are tens of millions lines of code out there written with this
assumption, and a comparatively simple change to gcc would prevent a
whole class of security problems in just about any Linux and BSD box out
there.

Auditing all this existing code out there for this specific
vulnerability is just hopeless, fixing it in gcc is the only right thing
to do.  And the fix would be LIA-1 compliant overflow behaviour.

A point that was often raised against making -fwrapv behaviour the
default was performance issues.  But neither reading the existing
thread, nor extensive discussion in the context of bug report 30475
showed benchmarks backing this argument, or any kind of theoretic
argument why breaking signedness overflow behaviour actually helps in
any optimization.

I then did some code reading, again see bug 30475.  And it turns out
that -fwrapv is implemented in a way that actually makes the reader
afraid that it does hurt performance.  As an example, the optimization
that is hurting us in the security context is folding of comparisons
based on integer range analysis.  The code for generating ranges
(fold-const.c, make_range()) for instance won't generate ranges that
properly respect overflow behaviour of signed ints.  Now instead of
fixing this, and generating proper ranges for signed ints when -fwrapv
is passed, the code continues to generate broken ranges, but turns off
all downstream optimizations depending on ranges if the expression is
signed.

There is a clear way to fix the performance issues of -fwrapv and still
make signed int overflow do the right thing.  I would like to appeal to
the gcc developers and the Steering Board to re-visit their position on
the issue, and take the route of making -fwrapv the default, and then
re-engineering it to do the right thing performance-wise.

Best regards,
Andreas