Bug 29433 - using boost::MPL requires lots of memory
Summary: using boost::MPL requires lots of memory
Status: RESOLVED FIXED
Alias: None
Product: gcc
Classification: Unclassified
Component: c++ (show other bugs)
Version: 4.0.3
: P3 normal
Target Milestone: 4.3.0
Assignee: Not yet assigned to anyone
URL:
Keywords: compile-time-hog, memory-hog
: 32290 (view as bug list)
Depends on: 30161
Blocks: 32052
  Show dependency treegraph
 
Reported: 2006-10-11 19:05 UTC by Siarhei Rachytski
Modified: 2007-06-12 11:29 UTC (History)
7 users (show)

See Also:
Host:
Target: i486-linux-gnu
Build:
Known to work:
Known to fail: 4.0.4 4.1.2 4.2.0
Last reconfirmed: 2006-10-11 21:56:21


Attachments
preprocessed source file (222.62 KB, application/zip)
2006-10-11 19:10 UTC, Siarhei Rachytski
Details
gcc -V output (372 bytes, text/plain)
2006-10-11 19:13 UTC, Siarhei Rachytski
Details
Command Line and Error Message (285 bytes, text/plain)
2006-10-11 19:14 UTC, Siarhei Rachytski
Details
preprocessed source file (version 2) (184.61 KB, application/octet-stream)
2006-10-12 09:51 UTC, Siarhei Rachytski
Details
preprocessed source file (version 3) (184.82 KB, application/octet-stream)
2006-10-12 16:50 UTC, Siarhei Rachytski
Details
testcase run through uninclude (87.08 KB, application/octet-stream)
2006-11-22 12:39 UTC, Richard Biener
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Siarhei Rachytski 2006-10-11 19:05:06 UTC
Here is a code which doesn't compile. It takes all swap space (around 2GB) and all system memory(about 1GB), compiles around 15-20 minutes and then crashes with the following message : "g++: Internal error: Killed (program cc1plus)"
Comment 1 Siarhei Rachytski 2006-10-11 19:10:16 UTC
Created attachment 12409 [details]
preprocessed source file

This is output of the compiler with the -save-temps switch. It's compressed 'cause there is a 1 MB limit on the attachement size.
Comment 2 Siarhei Rachytski 2006-10-11 19:13:28 UTC
Created attachment 12410 [details]
gcc -V output

version and configuration information on the used gcc.
Comment 3 Siarhei Rachytski 2006-10-11 19:14:20 UTC
Created attachment 12411 [details]
Command Line and Error Message
Comment 4 Andrew Pinski 2006-10-11 19:36:38 UTC
This takes 79% of my 2GB of memory.
Comment 5 Richard Biener 2006-10-11 21:56:21 UTC
Confirmed.  Uses a lot of memory.
Comment 6 Richard Biener 2006-10-12 08:20:21 UTC
Execution times (seconds)
 garbage collection    :   0.88 ( 1%) usr   0.00 ( 0%) sys   0.87 ( 1%) wall       0 kB ( 0%) ggc
 preprocessing         :   0.10 ( 0%) usr   0.04 ( 1%) sys   0.22 ( 0%) wall     796 kB ( 0%) ggc
 parser                :  17.90 (14%) usr   0.77 (15%) sys  19.39 (15%) wall  325217 kB (64%) ggc
 name lookup           :  94.51 (76%) usr   4.25 (84%) sys  98.43 (75%) wall  175474 kB (34%) ggc
 tree gimplify         :   0.07 ( 0%) usr   0.00 ( 0%) sys   0.04 ( 0%) wall    2548 kB ( 1%) ggc
 varconst              :  11.33 ( 9%) usr   0.01 ( 0%) sys  11.33 ( 9%) wall    2833 kB ( 1%) ggc
 symout                :   0.00 ( 0%) usr   0.00 ( 0%) sys   0.01 ( 0%) wall      32 kB ( 0%) ggc
 TOTAL                 : 124.80             5.08           130.45             508745 kB

and using peak 1.7GB ram (x86_64 with -m32) on the trunk.  I guess we're just
not collecting garbage during parsing at all.

Note that I get errors from compiling the testcase though:

[... very large template ...]
test_basic_metafunctions.hpp:176:   instantiated from here
../src/sd/../static_component.hpp:91: error: 'process_outputs' is not a member of 'mpl_::void_'

(4.1 also isn't happy with the testcase)

Can you by any chance provide a testcase that compiles ok with at least gcc 4.1.x?  (even better mainline...)  Thanks!

Profiling the bugger yields:

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls  Ks/call  Ks/call  name    
 12.44    178.30   178.30 121375686     0.00     0.00  comptypes
  9.29    311.48   133.18 218375227     0.00     0.00  dump_aggr_type
  8.69    436.02   124.54 155559998     0.00     0.00  comp_template_args
  8.59    559.19   123.17 620924334     0.00     0.00  pp_base_append_text
  7.85    671.72   112.53  1278486     0.00     0.00  retrieve_specialization
  6.92    770.99    99.27 173585104     0.00     0.00  template_args_equal
  6.54    864.80    93.81 462049326     0.00     0.00  pp_base_character
  5.05    937.24    72.44    67358     0.00     0.00  ht_lookup
  3.32    984.79    47.55 620924334     0.00     0.00  pp_base_string
  3.13   1029.71    44.92 186232097     0.00     0.00  dump_decl
  2.58   1066.65    36.95 218532269     0.00     0.00  pp_c_type_qualifier_list
  2.36   1100.46    33.81 16289532     0.00     0.00  dump_template_parms
  2.32   1133.71    33.25 404449103     0.00     0.00  dump_scope
  2.28   1166.40    32.69 218677178     0.00     0.00  dump_type
  1.95   1194.40    28.00 404642903     0.00     0.00  pp_c_identifier
  1.78   1219.92    25.53 218375227     0.00     0.00  class_key_or_enum_as_string
  1.38   1239.65    19.73 216176146     0.00     0.00  dump_template_argument
  1.03   1254.37    14.72  3919144     0.00     0.00  purpose_member
  0.88   1266.94    12.57 108004962     0.00     0.00  pp_base_last_position_in_text
  0.68   1276.74     9.80 162182109     0.00     0.00  pp_cxx_separate_with
  0.61   1285.53     8.80 202214181     0.00     0.00  pp_cxx_colon_colon
  0.41   1291.38     5.85 54002002     0.00     0.00  pp_cxx_end_template_argument_list
  0.41   1297.19     5.81  5800051     0.00     0.00  ggc_alloc_stat
  0.40   1302.92     5.73  4113746     0.00     0.00  tsubst
  0.37   1308.20     5.29 54002002     0.00     0.00  pp_cxx_begin_template_argument_list
  0.31   1312.58     4.38   301296     0.00     0.00  pp_base_emit_prefix
  0.30   1316.88     4.30   450268     0.00     0.00  gt_ggc_mx_lang_tree_node
  0.27   1320.81     3.93   302601     0.00     0.00  reinit_cxx_pp
  0.27   1324.73     3.92  6289194     0.00     0.00  ggc_set_mark
  0.26   1328.46     3.73       21     0.00     0.00  pp_base_newline
  0.25   1332.04     3.59   892391     0.00     0.00  lookup_template_class
  0.24   1335.43     3.39 14090587     0.00     0.00  pp_c_constant
  0.20   1338.34     2.91    67465     0.00     0.00  push_to_top_level
  0.20   1341.19     2.85  2159736     0.00     0.00  dfs_walk_all
  0.17   1343.61     2.43   935808     0.00     0.00  coerce_template_parms
  0.17   1346.03     2.42  1554163     0.00     0.00  lookup_field_1
  0.16   1348.27     2.24 10344444     0.00     0.00  cp_tree_equal
  0.15   1350.49     2.22   222604     0.00     0.00  ht_lookup_with_hash

Comment 7 Siarhei Rachytski 2006-10-12 09:51:54 UTC
Created attachment 12415 [details]
preprocessed source file (version 2)

The previous test case had common C++ error in it, which prevents it from being compiled properly. Try this one. It generates the same "internal error" message, but I think that it's free from the C++ errors. If not, I'll make some steps back and provide an example which also requires a lot of memory to compile, but compiles properly, without "internal error".
Comment 8 Richard Biener 2006-10-12 13:09:01 UTC
No success yet:

test_basic_metafunctions.hpp:176:   instantiated from here
../src/sd/../static_component.hpp:57: error: 'process_outputs' is not a member of 'mpl_::void_'

and other errors.
Comment 9 Siarhei Rachytski 2006-10-12 16:50:34 UTC
Created attachment 12420 [details]
preprocessed source file (version 3)

Compiles without errors on my laptop, but still eats a LOT of memory.
Comment 10 Siarhei Rachytski 2006-10-12 19:56:24 UTC
Subject: Re:  using boost::MPL requires lots of memory

Try preprocessed source file v3. It compiles without error on my
laptop, but still takes a LOT of memory and time to compile.

On 12 Oct 2006 13:09:02 -0000, rguenth at gcc dot gnu dot org
<gcc-bugzilla@gcc.gnu.org> wrote:
>
>
> ------- Comment #8 from rguenth at gcc dot gnu dot org  2006-10-12 13:09 -------
> No success yet:
>
> test_basic_metafunctions.hpp:176:   instantiated from here
> ../src/sd/../static_component.hpp:57: error: 'process_outputs' is not a member
> of 'mpl_::void_'
>
> and other errors.
>
>
> --
>
>
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29433
>
> ------- You are receiving this mail because: -------
> You reported the bug, or are watching the reporter.
>
Comment 11 Richard Biener 2006-10-13 10:58:00 UTC
That works.  Updated profile:

Flat profile:

Each sample counts as 0.01 seconds.
  %   cumulative   self              self     total           
 time   seconds   seconds    calls  Ks/call  Ks/call  name    
 11.26    131.84   131.84 77379135     0.00     0.00  comptypes
 11.05    261.22   129.38 153228836     0.00     0.00  dump_aggr_type
  7.47    348.70    87.48 441675354     0.00     0.00  pp_base_append_text
  7.06    431.38    82.68 101695144     0.00     0.00  comp_template_args
  6.90    512.09    80.71   870176     0.00     0.00  retrieve_specialization
  6.09    583.32    71.24 116984878     0.00     0.00  template_args_equal
  4.82    639.78    56.46 325283356     0.00     0.00  pp_base_character
  4.56    693.10    53.32    59058     0.00     0.00  ht_lookup
  3.63    735.58    42.48 441675354     0.00     0.00  pp_base_string
  3.30    774.18    38.60 287071962     0.00     0.00  dump_scope
  3.20    811.63    37.46 10634496     0.00     0.00  dump_template_parms
  2.99    846.65    35.02 133934524     0.00     0.00  dump_decl
  2.82    879.71    33.06 153423130     0.00     0.00  dump_type
  2.25    905.99    26.29 153344660     0.00     0.00  pp_c_type_qualifier_list
  1.97    929.03    23.04 152685422     0.00     0.00  dump_template_argument
  1.96    951.95    22.92 287215230     0.00     0.00  pp_c_identifier
  1.35    967.81    15.86  3824263     0.00     0.00  purpose_member
  1.34    983.49    15.68 153228836     0.00     0.00  class_key_or_enum_as_string
  0.77    992.52     9.03 115629446     0.00     0.00  pp_cxx_separate_with
  0.68   1000.52     8.00 144377450     0.00     0.00  pp_cxx_colon_colon
  0.60   1007.53     7.01 74128630     0.00     0.00  pp_base_last_position_in_text
  0.47   1013.00     5.47  4619635     0.00     0.00  ggc_alloc_stat
  0.43   1017.98     4.98 37063838     0.00     0.00  pp_cxx_end_template_argument_list
  0.42   1022.89     4.92  2734603     0.00     0.00  tsubst
  0.35   1026.98     4.09 37063838     0.00     0.00  pp_cxx_begin_template_argument_list
  0.35   1031.07     4.09   574289     0.00     0.00  gt_ggc_mx_lang_tree_node
  0.33   1034.88     3.81   218258     0.00     0.00  pp_base_emit_prefix
  0.31   1038.56     3.68  6251202     0.00     0.00  ggc_set_mark
  0.31   1042.21     3.66 10098142     0.00     0.00  pp_c_constant
  0.28   1045.54     3.33  1473688     0.00     0.00  dfs_walk_all
  0.28   1048.79     3.26   588277     0.00     0.00  lookup_template_class
  0.27   1051.95     3.16                             pp_base_newline
  0.23   1054.67     2.72 10081686     0.00     0.00  pp_c_integer_constant
  0.22   1057.22     2.55  1172353     0.00     0.00  lookup_field_1
  0.22   1059.75     2.53 10101968     0.00     0.00  dump_expr
  0.21   1062.20     2.45  4021108     0.00     0.00  dfs_access_in_type
  0.21   1064.64     2.44    48588     0.00     0.00  push_to_top_level
  0.20   1066.97     2.34  8942686     0.00     0.00  cp_tree_equal
  0.18   1069.12     2.15   615699     0.00     0.00  coerce_template_parms
  0.17   1071.14     2.03   219438     0.00     0.00  reinit_cxx_pp
  0.17   1073.11     1.97   214436     0.00     0.00  ht_lookup_with_hash
  0.16   1075.00     1.89   128624     0.00     0.00  get_expr_operands
  0.14   1076.60     1.60   609605     0.00     0.00  tsubst_aggr_type
  0.13   1078.18     1.58  2662882     0.00     0.00  store_binding
  0.13   1079.75     1.57   587010     0.00     0.00  instantiate_class_template
  0.12   1081.19     1.45        2     0.00     0.00  dump_function_decl
  0.12   1082.63     1.44   928019     0.00     0.00  tree_cons_stat
  0.12   1084.03     1.40   709984     0.00     0.00  tsubst_template_args
  0.12   1085.40     1.38  4639830     0.00     0.00  context_for_name_lookup
  0.10   1086.59     1.19  2314384     0.00     0.00  htab_find_with_hash
  0.09   1087.68     1.09   475135     0.00     0.00  lookup_fnfields_1
  0.09   1088.74     1.06    42858     0.00     0.00  finish_struct_1
  0.09   1089.78     1.04   695522     0.00     0.00  htab_find_slot_with_hash
  0.09   1090.81     1.03 10098142     0.00     0.00  pp_cxx_constant
  0.09   1091.83     1.02                            pp_c_conditional_expression
Comment 12 Richard Biener 2006-10-13 11:05:35 UTC
-O2 -ftime-report:

Execution times (seconds)
 parser                :  11.82 (14%) usr   0.54 (15%) sys  12.81 (14%) wall  254865 kB (62%) ggc
 name lookup           :  62.00 (74%) usr   3.04 (82%) sys  64.50 (73%) wall  123819 kB (30%) ggc
 varconst              :   7.37 ( 9%) usr   0.00 ( 0%) sys   7.50 ( 8%) wall    2369 kB ( 1%) ggc
 TOTAL                 :  84.28             3.71            88.39             411895 kB

! (profile was also -O2)

Not too much code is generated by the testcase (-O2):

> size main.o 
   text    data     bss     dec     hex filename
  21680       4      76   21760    5500 main.o
Comment 13 Siarhei Rachytski 2006-10-13 12:16:18 UTC
Subject: Re:  using boost::MPL requires lots of memory

But my main target is the previous testcase. It should at least
provide me a C++ error, but not crash with internal error.

On 13 Oct 2006 11:05:35 -0000, rguenth at gcc dot gnu dot org
<gcc-bugzilla@gcc.gnu.org> wrote:
>
>
> ------- Comment #12 from rguenth at gcc dot gnu dot org  2006-10-13 11:05 -------
> -O2 -ftime-report:
>
> Execution times (seconds)
>  parser                :  11.82 (14%) usr   0.54 (15%) sys  12.81 (14%) wall
> 254865 kB (62%) ggc
>  name lookup           :  62.00 (74%) usr   3.04 (82%) sys  64.50 (73%) wall
> 123819 kB (30%) ggc
>  varconst              :   7.37 ( 9%) usr   0.00 ( 0%) sys   7.50 ( 8%) wall
> 2369 kB ( 1%) ggc
>  TOTAL                 :  84.28             3.71            88.39
> 411895 kB
>
> ! (profile was also -O2)
>
> Not too much code is generated by the testcase (-O2):
>
> > size main.o
>   text    data     bss     dec     hex filename
>  21680       4      76   21760    5500 main.o
>
>
> --
>
>
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29433
>
> ------- You are receiving this mail because: -------
> You reported the bug, or are watching the reporter.
>
Comment 14 Richard Biener 2006-10-13 12:18:50 UTC
Sure, but the crash is due to gcc running out of memory.  And the third testcase is a good one to attack memory usage and compile-time problems on all of the three testcases.

But don't hold your breath...
Comment 15 Siarhei Rachytski 2006-10-13 12:20:57 UTC
Subject: Re:  using boost::MPL requires lots of memory

Ok, if there is anything else I could help, please mail me.

On 13 Oct 2006 12:18:51 -0000, rguenth at gcc dot gnu dot org
<gcc-bugzilla@gcc.gnu.org> wrote:
>
>
> ------- Comment #14 from rguenth at gcc dot gnu dot org  2006-10-13 12:18 -------
> Sure, but the crash is due to gcc running out of memory.  And the third
> testcase is a good one to attack memory usage and compile-time problems on all
> of the three testcases.
>
> But don't hold your breath...
>
>
> --
>
>
> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=29433
>
> ------- You are receiving this mail because: -------
> You reported the bug, or are watching the reporter.
>
Comment 16 Richard Biener 2006-10-13 13:19:36 UTC
The bootleneck is clearly using the diagnostic machinery for setting up DECL_NAME of the type decls.  I wonder if we cannot directly compute and store mangled names in DECL_NAME which would save both time and memory...

Any ideas?  Mark?  Jason?
Comment 17 Richard Biener 2006-11-22 12:39:09 UTC
Created attachment 12666 [details]
testcase run through uninclude

unincluded so it's 64bit clean also.
Comment 18 Richard Biener 2006-11-28 12:38:02 UTC
Most of the memory used is spent in the string pool:

String pool
entries         68980
identifiers     68980 (100.00%)
slots           131072
bytes           1012M (10M overhead)
table size      1024k
coll/search     0.7816
ins/search      0.2876
avg. entry      15394.37 bytes (+/- 68924.74)
longest entry   2130575

if you add -g to the compilation, dwarf2out.c happily xstrdup()s most of
them a few times and memory consumption goes up to 3.9GB.

All the stringpool handling and it's usage by classtype_mangled_name is
not very well optimized for such large strings (after all, we know the
string length there for example).
Comment 19 patchapp@dberlin.org 2006-11-28 16:21:32 UTC
Subject: Bug number PR29433

A patch for this bug has been added to the patch tracker.
The mailing list url for the patch is http://gcc.gnu.org/ml/gcc-patches/2006-11/msg01900.html
Comment 20 patchapp@dberlin.org 2006-11-30 16:15:25 UTC
Subject: Bug number PR29433

A patch for this bug has been added to the patch tracker.
The mailing list url for the patch is http://gcc.gnu.org/ml/gcc-patches/2006-11/msg02053.html
Comment 21 Richard Biener 2006-12-01 16:37:54 UTC
Subject: Bug 29433

Author: rguenth
Date: Fri Dec  1 16:37:38 2006
New Revision: 119404

URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=119404
Log:
2006-12-01  Richard Guenther  <rguenther@suse.de>

	PR c++/29433
	* dwarf2out.c (struct pubname_struct): Make name const.
	(add_pubtype): Do not xstrdup identifiers.

Modified:
    trunk/gcc/ChangeLog
    trunk/gcc/dwarf2out.c

Comment 22 Richard Biener 2006-12-01 16:38:41 UTC
Improved for the case of building with -g, which should now be as bad as with plain -O0.
Comment 23 Richard Biener 2007-02-23 14:21:32 UTC
Fixed.
Comment 24 Richard Biener 2007-06-12 11:29:39 UTC
*** Bug 32290 has been marked as a duplicate of this bug. ***