Re: Integer overflow in operator new

["J.C. Pizarro" <jcpiza at gmail dot com>]

2007/4/9, Ross Ridge <rridge@csclub.uwaterloo.ca> wrote:
> Florian Weimer writes:
> >Yeah, but that division is fairly expensive if it can't be performed
> >at compile time.  OTOH, if __compute_size is inlined in all places,
> >code size does increase somewhat.
>
> Well, I believe the assumption was that __compute_size would be inlined.
> If you want to minimize code size and avoid the division then a library
> function something like following might work:
>
>         void *__allocate_array(size_t num, size_t size, size_t max_num) {
>                 if (num > max_num)
>                         size = ~size_t(0);
>                 else
>                         size *= num;
>                 return operator new[](size);
>         }
>
> GCC would caclulate the constant "~size_t(0) / size" and pass it as the
> third argument.  You'ld be trading a multiply for a couple of constant
> outgoing arguments, so the code growth should be small.  Unfortunately,
> you'd be trading what in most cases is a fast shift and maybe add or
> two for slower multiply.
>
> So long as whatever switch is used to enable this check isn't on by
> default and its effect on code size and speed is documented, I don't
> think it matters that much what those effects are.  Anything that works
> should make the people concerned about security happy.   People more
> concerned with size or speed aren't going to enable this feature.
>
>                                         Ross Ridge

Hi Ross Ridge,

I tuned it a little bit.

-----------------------------------------------------------------------------
#include <stddef.h>

void *__allocate_array_of_RossRidge(size_t num, size_t size, size_t max_num) {

  if (num > max_num)
    size = ~size_t(0);
  else
    size *= num;
  return operator new[](size);
}

void *__allocate_array_of_JCPizarro(size_t num, size_t size, size_t max_num) {
  if (num > max_num) return operator new[](~size_t(0));
  return operator new[](size*num);
}

-----------------------------------------------------------------------------

_Z29__allocate_array_of_RossRidgejjj:
[ gcc v3.4.6 : 9 instructions ]
       movl    4(%esp), %edx
       cmpl    12(%esp), %edx
       movl    8(%esp), %eax
       orl     $-1, %eax
       imull   %edx, %eax
       pushl   %eax
       call    _Znaj
       popl    %edx
       ret

_Z29__allocate_array_of_RossRidgejjj:
[ gcc 4.1.3 20070326 (prerelease) : 8 instructions ]
	movl	4(%esp), %eax
	orl	$-1, %ecx
	cmpl	12(%esp), %eax
	movl	8(%esp), %edx
	movl	%edx, %ecx
	imull	%eax, %ecx
	movl	%ecx, 4(%esp)
	jmp	_Znaj

_Z29__allocate_array_of_JCPizarrojjj:
[ gcc 4.1.3 20070326 (prerelease) and gcc 3.4.6 : 7 instructions ]
	movl	4(%esp), %edx
	cmpl	12(%esp), %edx
	movl	8(%esp), %eax
	movl	$-1, 4(%esp)
	imull	%edx, %eax
	movl	%eax, 4(%esp)
	jmp	_Znaj

-----------------------------------------------------------------------------

J.C. Pizarro