Re: "virtual memory exhausted: Out of memory" with a large static array

[Michael Eager <eager at eagercon dot com>]

Pavel Saviankou wrote:
> Hi, 
>
> i would like to compile a programm, which contains an static array with a 
> large amount of structures(containts 9 doubles) as elements. I try it on two 
> ways. 


> and the array with the pointers on they:
> mystruct * array[823543] = { &struct1, &struct2, ...};
>
> Now i get the  "virtual memory exhausted: Out of memory" with both versions of 
> compiler and with both optimisations settings. 

There's nothing for the compiler to optimize.  No code is generated
when you initialize a large array.  All the compiler does is create
a huge assembly file which contains the values, which in turn creates
a huge object file.

> The source with the structures and array will be generated automatically. 
> With a small amount of structures - i have no problems, but i have to have a 
> large, more than 100 000 000 numbers of structures, amount of data. 

I would usually advise that you dynamically allocate memory using malloc
and read the data into this array.

But first, consider how much data you have.  Each of the elements in
your struct is 9 doubles, or 72 bytes.  When you attempt to create
100,000,000 elements each of which is 72 bytes, you will need 7.2Gb
memory to hold the array.  (Ignoring that the intermediate assembly
and object files will be larger than this.)  Do you really have this
much memory?  Does your operating system give this much memory to any
single process?

Whatever you are trying to do with this data will need to
be done incrementally.  Read in a chunk of data, process it
(perhaps partially) and move on to the next chunk.  If needed,
combine partial results to create your final result.

--
Michael Eager	 eager@eagercon.com
1960 Park Blvd., Palo Alto, CA 94306  650-325-8077