Collecting more frequently

["Martin v. Loewis" <martin at loewis dot home dot cs dot tu-berlin dot de>]

While compiling template-heavy C++, I found that memory consumption
dramatically goes up until the garbage collector is called the first
time. After that, it grows less intense. I figure the reason is that
g++ digests a lot of declarations, before emitting the first true
definition. Here is a trace (obtained from -Q) for stock 20000317:

{GC 5537k -> 2297k in 0.100}
{GC 5326k -> 4175k in 0.160}
{GC 87072k -> 19050k in 0.880}
{GC 24769k -> 19766k in 0.980}

Notice the peak of 87M allocated memory, which top(1) translated to
110M RSS. I've replaced in parse.y

extdefs:
		{ $<ttype>$ = NULL_TREE; }
	  lang_extdef
		{ $<ttype>$ = NULL_TREE; }
	| extdefs lang_extdef
		{ $<ttype>$ = NULL_TREE; ggc_collect(); }
	;

i.e. I now collect after each 'global' definition. With that, I got

{GC 5350k -> 2209k in 0.100}
{GC 5332k -> 4128k in 0.160}
{GC 5832k -> 5033k in 0.200} 
{GC 6555k -> 5177k in 0.200} 
{GC 6743k -> 5256k in 0.200} 
{GC 6886k -> 5321k in 0.220} 
{GC 6923k -> 5625k in 0.220} 
{GC 7318k -> 6587k in 0.260} 
{GC 8662k -> 6915k in 0.280} 
{GC 9078k -> 6981k in 0.280} 
{GC 9182k -> 7716k in 0.320} 
{GC 10048k -> 8826k in 0.360} 
{GC 11534k -> 9188k in 0.380} 
{GC 12022k -> 9459k in 0.380} 
{GC 12369k -> 10681k in 0.440} 
{GC 13928k -> 11051k in 0.480} 
{GC 14437k -> 11557k in 0.480} 
{GC 15107k -> 12249k in 0.520} 
{GC 15932k -> 13844k in 0.600} 
{GC 18038k -> 14589k in 0.640} 
{GC 18989k -> 14687k in 0.640} 
{GC 19175k -> 14779k in 0.640} 
{GC 19223k -> 15479k in 0.680} 
{GC 20246k -> 16372k in 0.700} 
{GC 21490k -> 16492k in 0.720} 
{GC 21698k -> 16932k in 0.720}
{GC 26499k -> 19050k in 0.820}
{GC 24769k -> 19766k in 0.900}

You'll notice that peak memory consumption is much lower now, and that
time spent in collection is higher. On my system, this is a good
trade-off, since a peak of 110M is close to the available main memory,
whereas the peak of 43M in the second case is no problem.

I propose that garbage collection happens more frequently in the
'parsing' stage of input file processing. Calling it as often as above
is perhaps not reasonable, since there is not that much new garbage
every time. So instead, I propose that the threshold is higher,
e.g. collection should occur after each extdef only when the memory
consumption has doubled or tripled since the last collection.

I'd be willing to work on a patch for that strategy if desired.

Regards,
Martin

P.S. In case anybody wants to experiment with my data, I attach the
source I've used below. It is from ORBacus 4.0b2.

GenCPPTieH.ii.bz2