Re: Security vulernarability or security feature?

["Prateek Saxena" <prateeksaxena at gmail dot com>]

On Thu, Apr 24, 2008 at 2:20 PM, Ralph Loader <suckfish@ihug.co.nz> wrote:
> > I am very interested in seeing how this optimization can remove
>  > arithmetic overflows.
>
>  int foo (char * buf, int n)
>  {
>         // buf+n may overflow of the programmer incorrectly passes
>         // a large value of n.  But recent versions of gcc optimise
>         // to 'n < 100', removing the overflow.
>         return buf + n < buf + 100;
>  }

This clearly is insecure coding. The optimization to replace "buf + n
< buf + 100" with "n < 100" assumes something about the value of buf.
I assume that the location of "buf", could change arbitrarily accross
platforms depending on the memory layout.
I ran foo as follows, getting different outputs :

int main()
{
   printf ("%d\n", foo (0xbffffffc, 0x40000010));
   return 0;
}

When "foo" is :
int foo (char * buf, int n)
{
      return buf + n < buf + 100;
}
Result : 1

When foo is (due to optimization, lets say) :
int foo (char * buf, int n)
{
         return n < 100;
}
Result : 0

When such assumptions are made ... the compiler may eliminate the bug
in some cases giving the programmer a false feeling that "Oh! My code
is bug free". The problem is that when the code is compiled on a
different platform, with different switches, the bug may reappear. I
just wanted to bring out the point about the assumption ... may be a
diagnostic should be issued. Or the compiler is smart enough to figure
out the values of "buf" and only optimise on cases which are safe.

>
>  Compiled on i386, gcc-4.3.0 with -O2 gives:
>
>  foo:
>         xorl    %eax, %eax
>         cmpl    $99, 8(%esp)
>         setle   %al
>         ret
>
>  E.g., calling foo with:
>
>  #include <stdio.h>
>  int main()
>  {
>     char buf[100];
>     printf ("%d\n", foo (buf, 1500000000));
>     return 0;
>  }
>
>  on my PC (where the stack is just below the 3Gig position).
>
>
>
>  > > Why is Cert advising people to avoid an optimisation that can ---
>  > > realistically, although probably rarely --- remove security
>  > > vulnerabilities?
>  > >
>  > If you are referring to VU#694123, this refers to an optimization
>
>  I'm talking about 162289.
>
>  Ralph.
>
>
>
>  > that removes checks pointer arithmetic wrapping.  The optimization
>  > doesn't actually eliminate the wrapping behavior; this still occurs.
>  > It does, however, eliminate certain kinds of checks (that depend upon
>  > undefined behavior).
>  >
>  > Thanks,
>  > rCs
>