[Bug c++/12850] memory consumption for heavy template instantiations tripled since 3.3

["steven at gcc dot gnu dot org" <gcc-bugzilla at gcc dot gnu dot org>]

------- Additional Comments From steven at gcc dot gnu dot org  2004-12-23 12:19 -------
The initial CP lexer bugger size is 10000:

#define CP_LEXER_BUFFER_SIZE 10000

That came in with the lex-all-ahead patch from Matt and Zack,
on 2004-09-20 (parser.c rev. 1.250 for the CVS history diggers)
but it seems a bit low to me if you're going to lex the whole
file up front.  I would not be surprised if the average C++
code with lots of templates has several 100,000 tokens...  Let
me see:
- preprocessed sources for generate.ii from PR8361, blank and
  pound lines stripped:  36200 lines
- an average of 7 tokens per line in the first 500 lines, let's
  assume that's a reasonable average for the whole file (it's
  easy to instrument g++ to get the exact number of tokens, if
  you want more accurate numbers ;-)

That makes it >250,000 tokens for this file.

Since we double the buffer, we have:
10,000 + 20,000 + 40,000 + 80,000 + 160,000 + 320,000 = 630000

That is the number of tokens we have allocate room for, with no
ggc-collect in the middle.  With ggc-page, which has power-of-2
based page sizes, it's safe to assume that each previous buffer
is too small to be reallocated, so a full new buffer is allocated
and the old one is memcpy-ed to the new one.  With checking off,
we ggc_free the old buffer, but with checking enabled we don't
so after finishing the whole lexing process, we have keep around
a buffer of ~380,000*sizeof(cp_token), so that's roughly 10MB
of memory we can't reclaim until the first ggc_collect call.

Maybe buffer should not be in GC memory at all?  We know the
exact live time of buffer, and as far as I can tell we never
ggc_collect while it is live.  According to the comments for
cp_lexer, "Tokens are never added to the cp_lexer after it is
created."  So it may be cheaper to have the buffer xmalloced,
and memcpy-ed to a buffer in GC space just before saving it
in the new cp_lexer object.

So two suggestions for a person who wants to make g++ a little
faster here:
- make CP_LEXER_BUFFER_SIZE larger.  To make it use pages more
  efficiently, look for some ratio of pagesize/(sizeof (cp_token))
- see buffer in parser.c:cp_lexer_new_main can be moved out of GC
  space as suggested above.



-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=12850