This is the mail archive of the gcc-patches@gcc.gnu.org mailing list for the GCC project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [PATCH] LTO streamer reorg - try to reduce WPA memory use


On Wed, 30 Jul 2014, Markus Trippelsdorf wrote:

> On 2014.07.30 at 10:31 +0200, Richard Biener wrote:
> > On Wed, Jul 30, 2014 at 7:51 AM, Markus Trippelsdorf
> > <markus@trippelsdorf.de> wrote:
> > > On 2014.07.29 at 15:10 +0200, Richard Biener wrote:
> > >> On Tue, 29 Jul 2014, Richard Biener wrote:
> > >>
> > >> >
> > >> > This re-organizes the LTO streamer to do compression transparently
> > >> > in the data-streamer routines (and disables section compression
> > >> > by defaulting to -flto-compression-level=0).  This avoids
> > >> > keeping the whole uncompressed sections in memory, only retaining
> > >> > the compressed ones.
> > >> >
> > >> > The downside is that we lose compression of at least the string
> > >> > parts (they are abusing the streaming interface quite awkwardly
> > >> > and doing random-accesses with offsets into the uncompressed
> > >> > section).  With a little bit of surgery we can get that back I
> > >> > think (but we'd have to keep the uncompressed piece in memory
> > >> > somewhere which means losing the memory use advantage).
> > >> >
> > >> > Very lightly tested sofar (running lto.exp).  I'll try a LTO
> > >> > bootstrap now.
> > >> >
> > >> > I wonder what the change is on WPA memory use for larger
> > >> > projects and what the effect on object file size is.
> > >>
> > >> Updated patch passing LTO bootstrap (one warning fix) and
> > >> with a memory leak fixed.
> > >
> > > Testing with Firefox is impossible at the moment because of PR61885.
> > > One thing I've noticed (before the ICE) is that virtual memory usage is
> > > very high:
> > >
> > > Address            Kbytes      RSS    Dirty  Mode  Mapping
> > > 0000000000400000    16344     9084        0  r-x-- lto1
> > > 00000000013f6000       36       36       28  rw--- lto1
> > > 00000000013ff000     1072      276      276  rw---   [ anon ]
> > > 00000000034aa000 10154940  1540384  1540384  rw---   [ anon ]
> > > 00002acf04af2000      136      136        0  r-x-- ld-2.19.90.so
> > > 00002acf04b14000       88       88       88  rw---   [ anon ]
> > > ...
> > > ----------------  -------  -------  -------
> > > total kB         12022060  3388396  3377708
> > 
> > Maybe there is still a memleak (just checked that LTOing int main() {}
> > doesn't leak).
> > 
> > Otherwise I forgot to enable compression at all ;)
> 
> He. Also parallel streaming stopped working.

Interesting.  Meanwhile after enabling compression the benefit of the
patch is the reduction in LTRANS file size (stage3 cc1) from
460MB to 178MB because we now compress here.

Object file size of stage3-gcc/*.o is increased to 540MB from 506MB
(note LTO bootstrap is with fat LTO objects, so the increase in
LTO IL size is bigger that it looks like).  I suppose mainly due
to no longer compressing strings.

I also can observe the appearant memleak - will try to hunt it down.

Richard.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]