512MB limit on arrays of structures containing arrays

[James A. Bednar]

I have an x86 machine with 1024MB of memory, and I can write a simple
program that uses up to about 890MB of that memory in a single
statically-allocated array with no problems.

However, if I instead use a 512MB or larger array of structures
containing arrays (see attached program), then compile it with g++,
the executable crashes as I initialize the elements of that array.  If
I compile with gcc or egcs instead of g++, the compiler aborts saying
that the "size of array ... is too large", which is decidedly more
friendly behavior.  However, this is still arguably an error since the
array is not in fact too large to fit in memory.  I have no problem if
I reduce the size of the array to an aggregate total less than 512MB,
if I split it into two similar arrays totalling up to 890MB, or if I
use a plain or multidimensional array of any size up to 890MB rather
than an array of structures containing arrays.

If GCC has a genuine 512MB limit on the size of such an array, and
there is an extraordinarily compelling reason to have such a limit,
then that limit should be documented and appropriate error messages
generated under both GCC and G++.  Otherwise, I consider the limit to
be a bug to be eradicated by e.g. using more bits for your array index
computations.

I have reproduced this problem only on one machine (the only one I
have with that much memory); it is a Pentium III 600Mhz running Red
Hat Linux 6.0 (kernel 2.2.12-25).  

The same problem is present under gcc egcs-2.91.66 19990314
(egcs-1.1.2 release) and gcc 2.95.1 19990816/Linux.  In gcc version
2.7.2.3 GCC acts just like G++ does in later versions, i.e. the
executable crashes rather than GCC displaying an error message.  Since
that error message was not in the 2.7.2.3 source code at all, the
check and the message were presumably added between 2.7.2.3 and
2.91.66 and maybe someone just forgot to do the same for G++.

Attached is source code for a simple test case that reproduces the
problem.  If the ARRAYDIM value is reduced by 1 to 905876, the program
works fine.  (906876*148*4 = 536870592, which is barely less than
512MB (536870912), while 906877*148*4 = 536871184, which is barely
greater than 512MB).

Hopefully you can at least put in an error message for G++, and it
would be really great if you could remove the 512MB artificial limit
altogether, since (as time goes on) surely more people will have
machines with this much memory.

Thanks,

Jim Bednar


###############################################################################

#define SIZE        148
#define ARRAYDIM 906877

typedef struct
{
  int array[ARRAYDIM];
} Str;

Str  map[SIZE];

int main(int argc, char **argv)
{
  unsigned int i,k;
  for(i=0; i<SIZE; i++)
    for(k=0; k<ARRAYDIM; k++)
      map[i].array[k]   = 1;
  return 0;
}

-------------------------------------------------------------------------------
G++ output:

g++ test.c -o test
./test
Segmentation fault (core dumped)
-------------------------------------------------------------------------------
GCC output: 

gcc test.c -o test
test.c:9: size of array `map' is too large

###############################################################################