This is the mail archive of the gcc-patches@gcc.gnu.org mailing list for the GCC project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [PATCH] LTO streamer reorg - try to reduce WPA memory use


On Wed, Jul 30, 2014 at 1:14 PM, Martin LiÅka <mliska@suse.cz> wrote:
>
> On 07/30/2014 11:41 AM, Richard Biener wrote:
>>
>> On Wed, 30 Jul 2014, Richard Biener wrote:
>>
>>> On Wed, Jul 30, 2014 at 7:51 AM, Markus Trippelsdorf
>>> <markus@trippelsdorf.de> wrote:
>>>>
>>>> On 2014.07.29 at 15:10 +0200, Richard Biener wrote:
>>>>>
>>>>> On Tue, 29 Jul 2014, Richard Biener wrote:
>>>>>
>>>>>> This re-organizes the LTO streamer to do compression transparently
>>>>>> in the data-streamer routines (and disables section compression
>>>>>> by defaulting to -flto-compression-level=0).  This avoids
>>>>>> keeping the whole uncompressed sections in memory, only retaining
>>>>>> the compressed ones.
>>>>>>
>>>>>> The downside is that we lose compression of at least the string
>>>>>> parts (they are abusing the streaming interface quite awkwardly
>>>>>> and doing random-accesses with offsets into the uncompressed
>>>>>> section).  With a little bit of surgery we can get that back I
>>>>>> think (but we'd have to keep the uncompressed piece in memory
>>>>>> somewhere which means losing the memory use advantage).
>>>>>>
>>>>>> Very lightly tested sofar (running lto.exp).  I'll try a LTO
>>>>>> bootstrap now.
>>>>>>
>>>>>> I wonder what the change is on WPA memory use for larger
>>>>>> projects and what the effect on object file size is.
>>>>>
>>>>> Updated patch passing LTO bootstrap (one warning fix) and
>>>>> with a memory leak fixed.
>>>>
>>>> Testing with Firefox is impossible at the moment because of PR61885.
>>>> One thing I've noticed (before the ICE) is that virtual memory usage is
>>>> very high:
>>>>
>>>> Address            Kbytes      RSS    Dirty  Mode  Mapping
>>>> 0000000000400000    16344     9084        0  r-x-- lto1
>>>> 00000000013f6000       36       36       28  rw--- lto1
>>>> 00000000013ff000     1072      276      276  rw---   [ anon ]
>>>> 00000000034aa000 10154940  1540384  1540384  rw---   [ anon ]
>>>> 00002acf04af2000      136      136        0  r-x-- ld-2.19.90.so
>>>> 00002acf04b14000       88       88       88  rw---   [ anon ]
>>>> ...
>>>> ----------------  -------  -------  -------
>>>> total kB         12022060  3388396  3377708
>>>
>>> Maybe there is still a memleak (just checked that LTOing int main() {}
>>> doesn't leak).
>>
>> Found it:
>>
>> Index: gcc/lto-section-in.c
>> ===================================================================
>> --- gcc/lto-section-in.c.orig   2014-07-30 12:40:27.950225826 +0200
>> +++ gcc/lto-section-in.c        2014-07-30 12:37:44.179237102 +0200
>> @@ -249,7 +249,7 @@ lto_destroy_simple_input_block (struct l
>>                                  struct lto_input_block *ib,
>>                                  const char *data, size_t len)
>>   {
>> -  free (ib);
>> +  delete ib;
>>     lto_free_section_data (file_data, section_type, NULL, data, len);
>>   }
>>   Richard.
>
> Hello,
>    there's memory/CPU usage for the patch. for both, I used sync and
> drop_caches.
>
> Url:
> https://drive.google.com/file/d/0B0pisUJ80pO1andOX19JMHV3LVE/edit?usp=sharing

Ok, it turns out setting -flto-compression-level to 0 doesn't really
short-circuit zlib for sections.  So the following does that the hard
but effective way.

Index: gcc/lto-section-out.c
===================================================================
--- gcc/lto-section-out.c.orig  2014-07-30 13:33:06.634008355 +0200
+++ gcc/lto-section-out.c       2014-07-30 13:29:19.468023995 +0200
@@ -80,7 +80,7 @@ lto_begin_section (const char *name, boo
      data is anything other than assembler output.  The effect here is that
      we get compression of IL only in non-ltrans object files.  */
   gcc_assert (compression_stream == NULL);
-  if (compress)
+  if (compress && 0)
     compression_stream = lto_start_compression (lto_append_data, NULL);
 }

Index: gcc/lto-section-in.c
===================================================================
--- gcc/lto-section-in.c.orig   2014-07-30 13:33:06.637008355 +0200
+++ gcc/lto-section-in.c        2014-07-30 13:31:57.329013126 +0200
@@ -153,7 +153,7 @@ lto_get_section_data (struct lto_file_de

   /* FIXME lto: WPA mode does not write compressed sections, so for now
      suppress uncompression if flag_ltrans.  */
-  if (!flag_ltrans)
+  if (!flag_ltrans && 0)
     {
       /* Create a mapping header containing the underlying data and length,
         and prepend this to the uncompression buffer.  The uncompressed data
@@ -200,7 +200,7 @@ lto_free_section_data (struct lto_file_d

   /* FIXME lto: WPA mode does not write compressed sections, so for now
      suppress uncompression mapping if flag_ltrans.  */
-  if (flag_ltrans)
+  if (flag_ltrans || 1)
     {
       (free_section_f) (file_data, section_type, name, data, len);
       return;

does that change anything?

Thanks,
Richard.

> Martin
>


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]