GCC memory consumption increased by recent patch!

[Jan Hubicka]

> I do not mean to be dense, but this is what I want.  Is there a point 
> during the compilation where the tree inlining has been done on all 
> functions, but the functions have not been dispatched to the rest of the 
> compilation?
Sorry for being confusing.  
At the moment the inlining happens just before all the other
optimizations so there is no place in the compiler, but I wanted to
mention that it is not conceptually dificult change.  For 4.0 however I
would preffer to not make this change and I would like to discuss some
alternatives before doing it for next release too.

We might want to enable it on development branch, but unfortunately I
even don't see resonable way how to evaulate it as currently our IPA
does not allow compilation of very large programs we are interested in.

> Gcc has a lot of memory problems but in my opinion most of them are 
> caused by grossly inefficient data structures rather than having too 
> many of some kind of structure. 

Definitly.
> 
> I understand that there is strong opposition to using an object oriented 
> language, but I also have a lot of experience building a compiler in an 
> object oriented language and there is a huge benefit from being able to 
> say that a tree is an abstract type with concrete subtypes that 
> represent each type of tree node in with just the right set of fields 
> for that type of tree node.
> 
> I believe that much of this can be accomplished with unions in c but I 
> am not a good enough c hacker to design such a system, or even to 
> comment on someone elses design.  However, I was appalled that there was 
> no way to add a field just for the static variables, which is the 
> correct way that much of this was implemented.  The trees and the 
> annotations of those trees are just too large and they are cross linked 
> so badly that there is no way to maintain any cache locality. 

Yes, this is anoying me too, this is mostly reason why I have made most
of datastructures used by cgraph separate from the tree nodes actually.
Zack has plans on redesigning the trees that should be able to make this
possible in easier way.  Definitly concerning the memory storage, one
can accomplish about the same effectivity with C unions and nested
structures as with OOo language, however the syntax is less handy.
> 
> I realize that to a certain extent I am barking at the moon.  The best 
> that one can do is to point out the obvious problems and hope that the 
> people who are the strong c hackers can get them fixed in a timely 
> manner.  At naturalbridge we compiled all of websphere in one 
> compilation (30000 classes) in less than 1 gb of memory with complete 
> cross class inlining. 

1GB of memory is still a lot even when it is much less than what GCC
would need.  I am pretty sure ICC can get around doing IPA without
actually allocating as much memory as all the .o files takes together.

Concerning GCC memory, we need about 700MB of memory to compile GCC
backend itself.   We do have serious problems then because the GGC
overhead increase out of resonable bounds (so we burn about 70% of
compilation time in GGC as long as I can reacall) and these things get
yet worse when machine goes to swap.

I will take a homework and look how other available production C
compilers (at least SGIpro) are dealing with this problem.

> If this is what I have to do for now, this is what I will do.  I had 
> just thought that there was more here.

For 4.0 lets don't change basic ordering of the passes - I have very bad
experiences with these kind of changes that should just work but usually
just break everything.  We should do more experimenting on
tree-profiling branch.  Perhaps I can get it working there as an
alternative option (-ftwo-pass-ipa) and we can see how much it will pay
back?

Honza
> 
> 
> >Honza
> > 
> >
> >>Kenny
> >>   
> >>
>