Re: WPA stream_out form & memory consumption

[Richard Biener <richard dot guenther at gmail dot com>]

On Tue, Mar 18, 2014 at 4:13 PM, Richard Biener
<richard.guenther@gmail.com> wrote:
> On Tue, Mar 18, 2014 at 4:09 PM, Martin LiÅka <mliska@suse.cz> wrote:
>> Hello,
>>    I've been compiling Chromium with LTO and I noticed that WPA stream_out
>> forks and do parallel:
>> http://gcc.gnu.org/ml/gcc-patches/2013-11/msg02621.html.
>>
>> I am unable to fit in 16GB memory: ld uses about 8GB and lto1 about 6GB.
>> When WPA start to fork, memory consumption increases so that lto1 is killed.
>> I would appreciate an --param option to disable this WPA fork. The number of
>> forks is taken from build system (-flto=9) which is fine for ltrans phase,
>> because LD releases aforementioned 8GB.
>>
>> What do you think about that?
>
> Hmm, what does cause the memory increase?  Are you eventually
> doing garbage collection after the forks in any of the processes?
> (just add a global flag may_not_gc_collect and set it to true before
> the first fork and assert in ggc_collect () that this flag isn't set)

We do.  At least here:

  lto_wpa_write_files ();
  if (!quiet_flag)
    fprintf (stderr, "\n");

  timevar_stop (TV_PHASE_STREAM_OUT);

  ggc_collect ();
  if (post_ipa_mem_report)
    {
      fprintf (stderr, "Memory consumption after IPA\n");
      dump_memory_report (false);
    }

try returning early before that ggc_collect () if none of the reports are
requested or move the ggc_collect () before lto_wpa_write_files
(which hopefully doesn't produce any gc garbage).

Richard.

> Richard.
>
>> Thanks,
>> Martin