US-CERT Vulnerability Note VU#162289

[Nicola Musatti]

Sorry to be so late in joining this discussion. I'm the person who 
originally notified Mark Mitchell about Microsoft's compiler performing 
this same optimization under certain conditions. Since mailing Mark on 
the subject I tried also VC++ 2008 and it behaves exactly like its 
predecessor. Here's my test program, directly derived from Mark's test case:

int f(char *buf, int len) {
   len = 1 << 30;
   if (buf + len < buf)
     return 1;


   return 0;
}

int main()
{
	char * b = "0123456789";
	for ( int l = 0; l < 1 << 30; ++l )
		f(b, l);
}

This is the command line shown by the IDE:
/Ox /GL /D "WIN32" /D "_CONSOLE" /D "_UNICODE" /D "UNICODE" /Gm /EHsc 
/MDd /GS- /Za /FAs /Fa"Debug\\" /Fo"Debug\\" /Fd"Debug\vc90.pdb" /W4 /c 
/Wp64 /Zi /TP .\cert.cpp

This is the assembler listing generated by VC++ 2008. VC++ 2005's 
listing differs only in the first line:

; Listing generated by Microsoft (R) Optimizing Compiler Version 
15.00.21022.08

	TITLE	d:\src\cert\cert.cpp
	.686P
	.XMM
	include listing.inc
	.model	flat


$SG-5	DB	'0123456789', 00H
PUBLIC	?f@@YAHPADH@Z					; f
; Function compile flags: /Ogtpy
; File d:\src\cert\cert.cpp
_TEXT	SEGMENT
?f@@YAHPADH@Z PROC					; f

; 2    :   len = 1 << 30;
; 3    :   if (buf + len < buf)
; 4    :     return 1;
; 5    :
; 6    :
; 7    :   return 0;

	xor	eax, eax

; 8    : }

	ret	0
?f@@YAHPADH@Z ENDP					; f
_TEXT	ENDS
PUBLIC	_main
; Function compile flags: /Ogtpy
_TEXT	SEGMENT
_main	PROC

; 12   : 	char * b = "0123456789";
; 13   : 	for ( int l = 0; l < 1 << 30; ++l )
; 14   : 		f(b, l);
; 15   : }

	xor	eax, eax
	ret	0
_main	ENDP
_TEXT	ENDS
END

I can make my project files available if anybody is interested.

I also tested CodeGear (Borland)'s C++ Builder 2007 compiler and as far 
as I can tell it doesn't perform this optimization. Maybe we should all 
switch to their product ;-)

Cheers,
Nicola Musatti
-- 
Nicola.Musatti <at> gmail <dot> com
Home: http://nicola.musatti.googlepages.com/home
Blog: http://wthwdik.wordpress.com/