operator new[] overflow (PR 19351)

[Chris Lattner]

On Nov 30, 2010, at 3:12 PM, Joe Buck wrote:

> On Tue, Nov 30, 2010 at 01:49:23PM -0800, Gabriel Dos Reis wrote:
>> The existing GCC behaviour is a bit more perverse than the
>> C malloc() case as in
>> 
>>       new T[n]
>> 
>> there is no multiplication that could be credited to careless programmer.
>> The multiplication is introduced by GCC.
> 
> ... which suggests strongly that GCC should fix it.  Too bad the ABI is
> frozen; if the internal ABI kept the two values (the size of the type, and
> the number of values) separate and passed two arguments to the allocation
> function, it would be easy to do the right thing (through bad_alloc if the
> multiplication overflows).

You don't need any ABI changes to support this.  For example, clang compiles:

int *foo(long X) {
  return new int[X];
}

into:

__Z3fool:                               ## @_Z3fool
Leh_func_begin0:
## BB#0:                                ## %entry
	movl	$4, %ecx
	movq	%rdi, %rax
	mulq	%rcx
	testq	%rdx, %rdx
	movq	$-1, %rdi
	cmoveq	%rax, %rdi
	jmp	__Znam

On overflow it just forces the size passed in to operator new to -1ULL, which throws bad_alloc.

-Chris