Security vulernarability or security feature?

[Ralph Loader]

Robert,

You have failed to answer my original question, and I think have failed
to understand the point of the example.

The example shows that what you are claiming is a vulnerability in
162289 is in fact a security feature.

>   that removes checks pointer arithmetic wrapping.

Just to be 100% clear, the optimisation I gave you an example of, and
which you think is "pretty cool" is precisely the same optimisation you
are complaining about in 162289, specifically, a pointer value may be
canceled from both sides of a comparison.

This is slightly confused by the fact that in the 162289 example, two
optimisations take place [gcc gurus please correct if I am wrong]:

(a) a pointer value is canceled from both sides of a comparison,
changing (buf+len<buf) to (len<0).  This is what changes the observable
behaviour of the code, and is what the debate is about.

(b) in the example given in 162289, (len<0) can then be evaluated at
compile time, removing the test entirely.   This does not change the
runtime behaviour of the code an is completely irrelevant.

To make it even clearer, we can disable optimisation (b) while leaving
optimisation (a), by making 'len' volatile:

int foo (char * buf)
{
    volatile int len = 1<<30;
    if (buf+len < buf)
	return 1;
    else
        return 0;
}

gcc then generates:

foo:
	subl	$16, %esp
	movl	$1073741824, 12(%esp)
	movl	12(%esp), %eax
	addl	$16, %esp
	shrl	$31, %eax
	ret

This has not completely removed the test, but when executed, it will
still always return 0, giving precisely the same run-time behaviour as
without the 'volatile'.

To re-iterate:

(a) Why does Cert advise against an optimisation that, under
the right circumstances, can remove examples of a historically
significant class of security holes.

(b) The example you claim is a length check in 162289 is not a length
check and does not support the conclusions you draw from it.

Ralph.


>  The optimization
> doesn't actually eliminate the wrapping behavior; this still occurs.
> It does, however, eliminate certain kinds of checks (that depend upon
> undefined behavior).
> 
> Thanks,
> rCs
>