[Bug lto/48978] excessive hash table allocation for large lto build

[hubicka at ucw dot cz]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=48978

--- Comment #1 from Jan Hubicka <hubicka at ucw dot cz> 2011-05-12 15:29:42 UTC ---
> I tried a large LTO build with gcc version 4.7.0 20110511 (experimental) (GCC) 
> on a 18GB machine. It ended with
> 
> lto1: out of memory allocating 8589934312 bytes after a total of 6827421696
> bytes
> 
> Since a 7+GB single malloc seems somewhat excessive I put a break point 
> on xmalloc_failed. It looks like the hash tables are growing
> too aggressively?

I think the problem was introduced by Richi's last commit:
2011-05-12  Richard Guenther  <rguenther@suse.de>

        * gimple.c (gtc_visit): Compare TREE_ADDRESSABLE, handle
        NULLPTR_TYPE similar to VOID_TYPE.  Defer type-leader lookup
        until after simple checks.
        (gimple_types_compatible_p): Likewise.
        (iterative_hash_gimple_type): Always hash pointer targets
        and function return and argument types.
        (iterative_hash_canonical_type): Do not hash TYPE_QUALS,
        hash TYPE_ALIGN.  Do not hash TYPE_MIN/MAX_VALUE.
        (gimple_canonical_types_compatible_p): Compare TREE_ADDRESSABLE,
        handle NULLPTR_TYPE similar to VOID_TYPE.  Handle non-aggregates
        completely in the simple compare section.
        (gimple_register_canonical_type): Query the cache again after
        registering.
So reverting this patch might get you past the problem.  Ysterday I was able
to build your testcase with 3GB of ram, today it got out of memory, too.

The hashtable is not expanding too irrationally, but it saves O(n^2)
information.
I would like to see it die for sure ;)

Honza