Re: A recent patch increased GCC's memory consumption in some cases!

[Jan Hubicka <jh at suse dot cz>]

Hi,
actually I can't make much sense of this memory usage increase. Looks
like tree node sizes increased
optabs.c:5369 (init_one_libfunc)                     118944:26.0% 0: 0.0%          0: 0.0%          0: 0.0%        531
optabs.c:5369 (init_one_libfunc)                     152928:31.1% 0: 0.0%          0: 0.0%      16992: 3.2%        531
531 libfuncs are now taking 152Kb instead of 118 comparing
http://www.suse.de/~gcctest/memory/results/200704211904/emptyO0.rep
and
http://www.suse.de/~gcctest/memory/results/200704221147/emptyO0.rep

Honza

> Hi,
> 
> I am a friendly script caring about memory consumption in GCC.  Please
> contact jh@suse.cz if something is going wrong.
> 
> Comparing memory consumption on compilation of combine.i, insn-attrtab.i,
> and generate-3.4.ii I got:
> 
> 
> comparing empty function compilation at -O0 level:
>   Peak amount of GGC memory allocated before garbage collecting increased from 2263k to 2322k, overall 2.61%
>   Peak amount of GGC memory still allocated after garbage collecting increased from 1952k to 1995k, overall 2.20%
>   Amount of produced GGC garbage increased from 446k to 479k, overall 7.35%
>   Amount of memory still referenced at the end of compilation increased from 2285k to 2371k, overall 3.77%
>     Overall memory needed: 7389k -> 7388k
>     Peak memory use before GGC: 2263k -> 2322k
>     Peak memory use after GGC: 1952k -> 1995k
>     Maximum of released memory in single GGC run: 311k -> 327k
>     Garbage: 446k -> 479k
>     Leak: 2285k -> 2371k
>     Overhead: 456k -> 516k
>     GGC runs: 3
> 
> comparing empty function compilation at -O0 -g level:
>   Peak amount of GGC memory allocated before garbage collecting increased from 2290k to 2350k, overall 2.62%
>   Peak amount of GGC memory still allocated after garbage collecting increased from 1979k to 2022k, overall 2.17%
>   Amount of produced GGC garbage increased from 449k to 482k, overall 7.30%
>   Amount of memory still referenced at the end of compilation increased from 2318k to 2404k, overall 3.70%
>     Overall memory needed: 7405k -> 7404k
>     Peak memory use before GGC: 2290k -> 2350k
>     Peak memory use after GGC: 1979k -> 2022k
>     Maximum of released memory in single GGC run: 311k -> 328k
>     Garbage: 449k -> 482k
>     Leak: 2318k -> 2404k
>     Overhead: 461k -> 520k
>     GGC runs: 3
> 
> comparing empty function compilation at -O1 level:
>   Peak amount of GGC memory allocated before garbage collecting increased from 2263k to 2322k, overall 2.61%
>   Peak amount of GGC memory still allocated after garbage collecting increased from 1952k to 1995k, overall 2.20%
>   Amount of produced GGC garbage increased from 452k to 485k, overall 7.22%
>   Amount of memory still referenced at the end of compilation increased from 2288k to 2374k, overall 3.76%
>     Overall memory needed: 7517k -> 7520k
>     Peak memory use before GGC: 2263k -> 2322k
>     Peak memory use after GGC: 1952k -> 1995k
>     Maximum of released memory in single GGC run: 311k -> 327k
>     Garbage: 452k -> 485k
>     Leak: 2288k -> 2374k
>     Overhead: 457k -> 516k
>     GGC runs: 4
> 
> comparing empty function compilation at -O2 level:
>   Peak amount of GGC memory allocated before garbage collecting increased from 2263k to 2323k, overall 2.65%
>   Peak amount of GGC memory still allocated after garbage collecting increased from 1952k to 1995k, overall 2.20%
>   Amount of produced GGC garbage increased from 455k to 488k, overall 7.18%
>   Amount of memory still referenced at the end of compilation increased from 2288k to 2374k, overall 3.76%
>     Overall memory needed: 7525k -> 7528k
>     Peak memory use before GGC: 2263k -> 2323k
>     Peak memory use after GGC: 1952k -> 1995k
>     Maximum of released memory in single GGC run: 311k -> 328k
>     Garbage: 455k -> 488k
>     Leak: 2288k -> 2374k
>     Overhead: 457k -> 517k
>     GGC runs: 4
> 
> comparing empty function compilation at -O3 level:
>   Peak amount of GGC memory allocated before garbage collecting increased from 2263k to 2323k, overall 2.65%
>   Peak amount of GGC memory still allocated after garbage collecting increased from 1952k to 1995k, overall 2.20%
>   Amount of produced GGC garbage increased from 455k to 488k, overall 7.18%
>   Amount of memory still referenced at the end of compilation increased from 2288k to 2374k, overall 3.76%
>     Overall memory needed: 7525k -> 7528k
>     Peak memory use before GGC: 2263k -> 2323k
>     Peak memory use after GGC: 1952k -> 1995k
>     Maximum of released memory in single GGC run: 311k -> 328k
>     Garbage: 455k -> 488k
>     Leak: 2288k -> 2374k
>     Overhead: 457k -> 517k
>     GGC runs: 4
> 
> comparing combine.c compilation at -O0 level:
>   Peak amount of GGC memory allocated before garbage collecting increased from 9264k to 9392k, overall 1.38%
>   Peak amount of GGC memory still allocated after garbage collecting increased from 8852k to 8982k, overall 1.47%
>   Amount of memory still referenced at the end of compilation increased from 6580k to 6873k, overall 4.44%
>     Overall memory needed: 17729k -> 17956k
>     Peak memory use before GGC: 9264k -> 9392k
>     Peak memory use after GGC: 8852k -> 8982k
>     Maximum of released memory in single GGC run: 2577k -> 2565k
>     Garbage: 37096k -> 36972k
>     Leak: 6580k -> 6873k
>     Overhead: 5053k -> 5163k
>     GGC runs: 282 -> 277
> 
> comparing combine.c compilation at -O0 -g level:
>   Peak amount of GGC memory allocated before garbage collecting increased from 10880k to 11011k, overall 1.20%
>   Peak amount of GGC memory still allocated after garbage collecting increased from 10512k to 10642k, overall 1.24%
>   Amount of memory still referenced at the end of compilation increased from 9465k to 9765k, overall 3.17%
>     Overall memory needed: 19889k -> 20004k
>     Peak memory use before GGC: 10880k -> 11011k
>     Peak memory use after GGC: 10512k -> 10642k
>     Maximum of released memory in single GGC run: 2351k -> 2343k
>     Garbage: 37693k -> 37568k
>     Leak: 9465k -> 9765k
>     Overhead: 5759k -> 5867k
>     GGC runs: 270 -> 269
> 
> comparing combine.c compilation at -O1 level:
>   Peak amount of GGC memory allocated before garbage collecting increased from 19347k to 19410k, overall 0.33%
>   Peak amount of GGC memory still allocated after garbage collecting increased from 19155k to 19215k, overall 0.31%
>   Amount of memory still referenced at the end of compilation increased from 6623k to 6911k, overall 4.35%
>     Overall memory needed: 33953k -> 33984k
>     Peak memory use before GGC: 19347k -> 19410k
>     Peak memory use after GGC: 19155k -> 19215k
>     Maximum of released memory in single GGC run: 2273k -> 2261k
>     Garbage: 51518k -> 51373k
>     Leak: 6623k -> 6911k
>     Overhead: 5991k -> 6064k
>     GGC runs: 348 -> 344
> 
> comparing combine.c compilation at -O2 level:
>   Peak amount of GGC memory allocated before garbage collecting increased from 19415k to 19473k, overall 0.30%
>   Peak amount of GGC memory still allocated after garbage collecting increased from 19213k to 19278k, overall 0.34%
>   Amount of produced GGC garbage increased from 67769k to 67933k, overall 0.24%
>   Amount of memory still referenced at the end of compilation increased from 6741k to 7030k, overall 4.29%
>     Overall memory needed: 38337k -> 38364k
>     Peak memory use before GGC: 19415k -> 19473k
>     Peak memory use after GGC: 19213k -> 19278k
>     Maximum of released memory in single GGC run: 2200k -> 2188k
>     Garbage: 67769k -> 67933k
>     Leak: 6741k -> 7030k
>     Overhead: 8056k -> 8155k
>     GGC runs: 399 -> 398
> 
> comparing combine.c compilation at -O3 level:
>   Peak amount of GGC memory allocated before garbage collecting increased from 19714k to 19801k, overall 0.44%
>   Peak amount of GGC memory still allocated after garbage collecting increased from 19380k to 19478k, overall 0.51%
>   Amount of produced GGC garbage increased from 92940k to 93626k, overall 0.74%
>   Amount of memory still referenced at the end of compilation increased from 6847k to 7131k, overall 4.14%
>     Overall memory needed: 42729k -> 42768k
>     Peak memory use before GGC: 19714k -> 19801k
>     Peak memory use after GGC: 19380k -> 19478k
>     Maximum of released memory in single GGC run: 3715k -> 3723k
>     Garbage: 92940k -> 93626k
>     Leak: 6847k -> 7131k
>     Overhead: 11444k -> 11587k
>     GGC runs: 427 -> 426
> 
> comparing insn-attrtab.c compilation at -O0 level:
>   Amount of memory still referenced at the end of compilation increased from 9583k to 9828k, overall 2.56%
>     Overall memory needed: 102849k -> 102696k
>     Peak memory use before GGC: 68627k -> 68320k
>     Peak memory use after GGC: 44730k -> 44619k
>     Maximum of released memory in single GGC run: 36429k -> 36234k
>     Garbage: 130923k -> 130216k
>     Leak: 9583k -> 9828k
>     Overhead: 16927k -> 16822k
>     GGC runs: 212 -> 211
> 
> comparing insn-attrtab.c compilation at -O0 -g level:
>   Amount of memory still referenced at the end of compilation increased from 11067k to 11536k, overall 4.24%
>     Overall memory needed: 104357k -> 103940k
>     Peak memory use before GGC: 69788k -> 69481k
>     Peak memory use after GGC: 45998k -> 45886k
>     Maximum of released memory in single GGC run: 36429k -> 36234k
>     Garbage: 132399k -> 131441k
>     Leak: 11067k -> 11536k
>     Overhead: 17324k -> 17218k
>     GGC runs: 210 -> 209
> 
> comparing insn-attrtab.c compilation at -O1 level:
>   Peak amount of GGC memory still allocated after garbage collecting increased from 69350k to 69844k, overall 0.71%
>   Amount of memory still referenced at the end of compilation increased from 9405k to 9650k, overall 2.61%
>     Overall memory needed: 121853k -> 120452k
>     Peak memory use before GGC: 73426k -> 71729k
>     Peak memory use after GGC: 69350k -> 69844k
>     Maximum of released memory in single GGC run: 32798k -> 32923k
>     Garbage: 227633k -> 225564k
>     Leak: 9405k -> 9650k
>     Overhead: 27090k -> 26613k
>     GGC runs: 226 -> 225
> 
> comparing insn-attrtab.c compilation at -O2 level:
>   Peak amount of GGC memory allocated before garbage collecting run decreased from 74717k to 72376k, overall -3.23%
>   Peak amount of GGC memory still allocated after garbage collecting increased from 69433k to 69922k, overall 0.70%
>   Amount of memory still referenced at the end of compilation increased from 9402k to 9648k, overall 2.61%
>     Overall memory needed: 187921k -> 183928k
>     Peak memory use before GGC: 74717k -> 72376k
>     Peak memory use after GGC: 69433k -> 69922k
>     Maximum of released memory in single GGC run: 29993k -> 29816k
>     Garbage: 263315k -> 261120k
>     Leak: 9402k -> 9648k
>     Overhead: 32727k -> 32159k
>     GGC runs: 245
> 
> comparing insn-attrtab.c compilation at -O3 level:
>   Peak amount of GGC memory still allocated after garbage collecting increased from 78009k to 80330k, overall 2.98%
>   Amount of memory still referenced at the end of compilation increased from 9406k to 9652k, overall 2.61%
>     Overall memory needed: 196985k -> 200912k
>     Peak memory use before GGC: 84750k -> 84491k
>     Peak memory use after GGC: 78009k -> 80330k
>     Maximum of released memory in single GGC run: 30928k -> 30750k
>     Garbage: 292401k -> 292453k
>     Leak: 9406k -> 9652k
>     Overhead: 33464k -> 33422k
>     GGC runs: 246
> 
> comparing Gerald's testcase PR8361 compilation at -O0 level:
>   Peak amount of GGC memory allocated before garbage collecting increased from 90233k to 90673k, overall 0.49%
>   Peak amount of GGC memory still allocated after garbage collecting increased from 89339k to 89775k, overall 0.49%
>   Amount of memory still referenced at the end of compilation increased from 49198k to 50285k, overall 2.21%
>     Overall memory needed: 147064k -> 147572k
>     Peak memory use before GGC: 90233k -> 90673k
>     Peak memory use after GGC: 89339k -> 89775k
>     Maximum of released memory in single GGC run: 17774k -> 18043k
>     Garbage: 207965k -> 206332k
>     Leak: 49198k -> 50285k
>     Overhead: 23912k -> 23772k
>     GGC runs: 410 -> 409
> 
> comparing Gerald's testcase PR8361 compilation at -O0 -g level:
>   Peak amount of GGC memory allocated before garbage collecting increased from 102839k to 103360k, overall 0.51%
>   Peak amount of GGC memory still allocated after garbage collecting increased from 101821k to 102332k, overall 0.50%
>   Amount of memory still referenced at the end of compilation increased from 72513k to 73600k, overall 1.50%
>     Overall memory needed: 164824k -> 165420k
>     Peak memory use before GGC: 102839k -> 103360k
>     Peak memory use after GGC: 101821k -> 102332k
>     Maximum of released memory in single GGC run: 18128k -> 18488k
>     Garbage: 214474k -> 212820k
>     Leak: 72513k -> 73600k
>     Overhead: 29808k -> 29648k
>     GGC runs: 384 -> 381
> 
> comparing Gerald's testcase PR8361 compilation at -O1 level:
>   Amount of memory still referenced at the end of compilation increased from 50289k to 51180k, overall 1.77%
>     Overall memory needed: 145840k -> 145004k
>     Peak memory use before GGC: 101915k -> 101304k
>     Peak memory use after GGC: 100904k -> 100296k
>     Maximum of released memory in single GGC run: 17235k -> 17230k
>     Garbage: 338299k -> 335726k
>     Leak: 50289k -> 51180k
>     Overhead: 29905k -> 28502k
>     GGC runs: 524
> 
> comparing Gerald's testcase PR8361 compilation at -O2 level:
>   Amount of produced GGC garbage increased from 382634k to 383438k, overall 0.21%
>   Amount of memory still referenced at the end of compilation increased from 51012k to 51899k, overall 1.74%
>     Overall memory needed: 151192k -> 151320k
>     Peak memory use before GGC: 102520k -> 101978k
>     Peak memory use after GGC: 101504k -> 100939k
>     Maximum of released memory in single GGC run: 17232k -> 17227k
>     Garbage: 382634k -> 383438k
>     Leak: 51012k -> 51899k
>     Overhead: 35178k -> 33973k
>     GGC runs: 571 -> 573
> 
> comparing Gerald's testcase PR8361 compilation at -O3 level:
>   Amount of produced GGC garbage increased from 412662k to 415305k, overall 0.64%
>   Amount of memory still referenced at the end of compilation increased from 51297k to 52171k, overall 1.70%
>     Overall memory needed: 153192k -> 152272k
>     Peak memory use before GGC: 104339k -> 104108k
>     Peak memory use after GGC: 103298k -> 103021k
>     Maximum of released memory in single GGC run: 17648k -> 17595k
>     Garbage: 412662k -> 415305k
>     Leak: 51297k -> 52171k
>     Overhead: 37505k -> 36360k
>     GGC runs: 588 -> 591
> 
> comparing PR rtl-optimization/28071 testcase compilation at -O0 level:
>   Amount of memory still referenced at the end of compilation increased from 7613k to 7721k, overall 1.42%
>     Overall memory needed: 244781k -> 244832k
>     Peak memory use before GGC: 80962k -> 81016k
>     Peak memory use after GGC: 58701k -> 58755k
>     Maximum of released memory in single GGC run: 44134k -> 44133k
>     Garbage: 142205k -> 142235k
>     Leak: 7613k -> 7721k
>     Overhead: 24559k -> 24633k
>     GGC runs: 79
> 
> comparing PR rtl-optimization/28071 testcase compilation at -O0 -g level:
>   Amount of memory still referenced at the end of compilation increased from 9380k to 9489k, overall 1.16%
>     Overall memory needed: 245593k -> 245636k
>     Peak memory use before GGC: 81609k -> 81663k
>     Peak memory use after GGC: 59347k -> 59401k
>     Maximum of released memory in single GGC run: 44122k
>     Garbage: 142428k -> 142454k
>     Leak: 9380k -> 9489k
>     Overhead: 25054k -> 25129k
>     GGC runs: 89 -> 88
> 
> comparing PR rtl-optimization/28071 testcase compilation at -O1 level:
>     Overall memory needed: 248201k -> 248028k
>     Peak memory use before GGC: 84261k -> 83890k
>     Peak memory use after GGC: 74847k -> 74753k
>     Maximum of released memory in single GGC run: 36149k -> 36146k
>     Garbage: 221602k -> 221295k
>     Leak: 20856k -> 20557k
>     Overhead: 30442k -> 30206k
>     GGC runs: 81
> 
> comparing PR rtl-optimization/28071 testcase compilation at -O2 level:
>     Overall memory needed: 275933k -> 275500k
>     Peak memory use before GGC: 79833k -> 79740k
>     Peak memory use after GGC: 74847k -> 74754k
>     Maximum of released memory in single GGC run: 33439k -> 33435k
>     Garbage: 228323k -> 228016k
>     Leak: 20946k -> 20647k
>     Overhead: 32463k -> 32226k
>     GGC runs: 91
> 
> comparing PR rtl-optimization/28071 testcase compilation at -O3 -fno-tree-pre -fno-tree-fre level:
>   Amount of produced GGC garbage increased from 370042k to 375537k, overall 1.49%
>   Amount of memory still referenced at the end of compilation decreased from 46312k to 44591k, overall -3.86%
>     Overall memory needed: 1269041k -> 1267628k
>     Peak memory use before GGC: 201851k -> 198185k
>     Peak memory use after GGC: 190309k -> 186643k
>     Maximum of released memory in single GGC run: 80677k
>     Garbage: 370042k -> 375537k
>     Leak: 46312k -> 44591k
>     Overhead: 48588k -> 47760k
>     GGC runs: 70
> 
> Head of the ChangeLog is:
> 
> --- /usr/src/SpecTests/sandbox-britten-memory/x86_64/mem-result/ChangeLog	2007-04-21 19:11:23.000000000 +0000
> +++ /usr/src/SpecTests/sandbox-britten-memory/gcc/gcc/ChangeLog	2007-04-22 11:54:05.000000000 +0000
> @@ -1,3 +1,77 @@
> +2007-04-22  Richard Guenther  <rguenther@suse.de>
> +
> +	PR tree-optimization/29789
> +	* tree-ssa-loop-im.c (stmt_cost): Adjust cost of shifts.
> +	(rewrite_reciprocal): New helper split out from
> +	determine_invariantness_stmt.
> +	(rewrite_bittest): Likewise.
> +	(determine_invariantness_stmt): Rewrite (A >> B) & 1 to
> +	A & (1 << B) if (1 << B) is loop invariant but (A >> B)
> +	is not.
> +
> +2007-04-22  Revital Eres  <eres@il.ibm.com>
> +
> +        * loop-unroll.c (var_to_expand): New field to support also
> +        insns of the form x = something + x.
> +        (analyze_insn_to_expand_var): Use it.
> +        (expand_var_during_unrolling): Likewise.
> +
> +2007-04-21  Zdenek Dvorak  <dvorakz@suse.cz>
> +
> +	* predict.c: Include pointer-set.h.
> +	(bb_predictions): New variable.
> +	(tree_predicted_by_p, tree_predict_edge,
> +	remove_predictions_associated_with_edge): Use bb_predictions map
> +	instead of bb->predictions.
> +	(clear_bb_predictions, assert_is_empty): New functions.
> +	(combine_predictions_for_bb): Use bb_predictions map.  Call
> +	clear_bb_predictions.
> +	(tree_estimate_probability): Create and free bb_predictions map.
> +	* Makefile.in (predict.o): Add pointer-set.h dependency.
> +	* basic-block.h (struct basic_block_def): Remove predictions
> +	field.
> +	* cfgrtl.c (rtl_verify_flow_info_1): Do not check bb->predictions.
> +
> +2007-04-21  Kaz Kojima  <kkojima@gcc.gnu.org>
> +
> +	PR target/31480
> +	* config/sh/sh.md (length): Check if prev_nonnote_insn (insn)
> +	is null.
> +
> +2007-04-21  Andrew Pinski  <andrew_pinski@playstation.sony.com>
> +
> +	* timevar.c (timevar_print): Change reference of --disable-checking to
> +	--enable-checking=release.  Also warn if assert checking is disabled.
> +
> +2007-04-21  Andrew Pinski  <andrew_pinski@playstation.sony.com>
> +
> +	PR C/30265
> +	* c-gimplifier.c (gimplify_compound_literal_expr): Mark the
> +	decl as addressable if the compound literal was marked as
> +	addressable.
> +	Mark the decl as a gimple register if it is a complex or
> +	vector decl and does not live in memory.
> +
> +2007-04-21  Andrew Pinski  <andrew_pinski@playstation.sony.com>
> +
> +	* tree.h (GIMPLE_TUPLE_P): Also true for PHI_NODEs.
> +	(GENERIC_NEXT): New function macro.
> +	(PHI_CHAIN): Use phi_node's new chain variable.
> +	(tree_phi_node): Change tree_common to tree_base
> +	and add the chain field.
> +	* tree-phinodes.c (make_phi_node): Don't set
> +	TREE_TYPE on the new node.
> +	* c-decl.c (lang_tree_node): Use GENERIC_NEXT
> +	instead of checking GIMPLE_TUPLE_P in chain_next.
> +	* tree-vect-transform.c
> +	(get_initial_def_for_induction): Look at
> +	PHI_RESULT_TREE for the type of the phi node.
> +	(update_vuses_to_preheader): Use PHI_CHAIN
> +	instead of TREE_CHAIN on the phi node.
> +	* tree-ssa-structalias.c (compute_points_to_sets):
> +	Likewise.
> +	(ipa_pta_execute): Likewise.
> +
>  2007-04-21  Richard Guenther  <rguenther@suse.de>
>  
>  	PR middle-end/31136
> --- /usr/src/SpecTests/sandbox-britten-memory/x86_64/mem-result/ChangeLog.cp	2007-04-18 07:23:05.000000000 +0000
> +++ /usr/src/SpecTests/sandbox-britten-memory/gcc/gcc/cp/ChangeLog	2007-04-22 11:54:05.000000000 +0000
> @@ -1,3 +1,8 @@
> +2007-04-21  Andrew Pinski  <andrew_pinski@playstation.sony.com>
> +
> +	* cp-tree.h (lang_tree_node): Use GENERIC_NEXT
> +	instead of checking GIMPLE_STMT_P in chain_next.
> +
>  2007-04-17  Mark Mitchell  <mark@codesourcery.com>
>  
>  	PR c++/31513
> 
> 
> The results can be reproduced by building a compiler with
> 
> --enable-gather-detailed-mem-stats targetting x86-64
> 
> and compiling preprocessed combine.c or testcase from PR8632 with:
> 
> -fmem-report --param=ggc-min-heapsize=1024 --param=ggc-min-expand=1 -Ox -Q
> 
> The memory consumption summary appears in the dump after detailed listing
> of the places they are allocated in.  Peak memory consumption is actually
> computed by looking for maximal value in {GC XXXX -> YYYY} report.
> 
> Your testing script.