Bug 34898 - Excessive memory consumption during compilation
Summary: Excessive memory consumption during compilation
Status: RESOLVED WONTFIX
Alias: None
Product: gcc
Classification: Unclassified
Component: ada (show other bugs)
Version: 4.3.0
: P3 normal
Target Milestone: ---
Assignee: Not yet assigned to anyone
URL:
Keywords: memory-hog
Depends on:
Blocks:
 
Reported: 2008-01-21 07:22 UTC by Oliver Kellogg
Modified: 2011-05-12 05:44 UTC (History)
2 users (show)

See Also:
Host: i686-pc-linux-gnu
Target:
Build:
Known to work:
Known to fail:
Last reconfirmed: 2008-05-25 18:56:20


Attachments
source code for producing the bug (482.06 KB, application/x-gtar)
2008-01-21 07:30 UTC, Oliver Kellogg
Details
statistics output from gnat1 on pkg001u.adb without aggregate assignments (777 bytes, text/plain)
2008-05-25 13:31 UTC, Oliver Kellogg
Details
statistics output from gnat1 on pkg001u.adb with one assignment (857 bytes, text/plain)
2008-05-25 13:38 UTC, Oliver Kellogg
Details
gnat1 (trunk r135848) output from -fmem-report, no aggregate assignments (11.70 KB, text/plain)
2008-05-25 18:12 UTC, Oliver Kellogg
Details
same as above but with assignments in pkg001u.adb lines 296 and 377 enabled (13.57 KB, text/plain)
2008-05-25 18:17 UTC, Oliver Kellogg
Details
att15682 was incorrect, two assignments already exhaust the memory. memreport for _one_ assignmt. (13.55 KB, text/plain)
2008-05-25 18:43 UTC, Oliver Kellogg
Details
instrumentation of build_simple_component_ref (for analysis only) (1.44 KB, patch)
2008-06-01 18:10 UTC, Oliver Kellogg
Details | Diff
console output from gnat1 with the above instrumentation (53.62 KB, application/x-gzip)
2008-06-01 18:28 UTC, Oliver Kellogg
Details
output from -fdump-tree-original of gnat1 compiling pkg001u.adb (81.30 KB, application/x-gzip)
2008-06-02 19:16 UTC, Oliver Kellogg
Details
regenerated statistics: trunk r139367 gnat1-gnat95 -fmem-report -fdump-tree-all pkg001u.adb (15.54 KB, text/plain)
2008-08-21 05:41 UTC, Oliver Kellogg
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Oliver Kellogg 2008-01-21 07:22:17 UTC
For reference, here's a compile with gcc-3.3.5:

$ gcc -c -v pkg001u.adb
Reading specs from /usr/lib/gcc-lib/i586-suse-linux/3.3.5/specs
Configured with: ../configure --enable-threads=posix --prefix=/usr --with-local-prefix=/usr/local --infodir=/usr/share/info --mandir=/usr/share/man --enable-languages=c,c++,f77,objc,java,ada --disable-checking --libdir=/usr/lib --enable-libgcj --with-slibdir=/lib --with-system-zlib --enable-shared --enable-__cxa_atexit i586-suse-linux
Thread model: posix
gcc version 3.3.5 20050117 (prerelease) (SUSE Linux)
 /usr/lib/gcc-lib/i586-suse-linux/3.3.5/gnat1 -quiet -dumpbase pkg001u.adb pkg001u.adb -o /tmp/cckqHQka.s

The file compiles successfully and 'top' shows a peak RES requirement of 77 Meg using no VIRT.
Thus let's limit the VIRT to a half gig,

$ ulimit -v 512000

and compile with a newer gcc:

$ gcc -c -v -gnat95 pkg001u.adb
Using built-in specs.
Target: i686-pc-linux-gnu
Configured with: ../../SOURCES/gcc/configure --prefix=/opt/gccsnap --enable-debug --enable-languages=c,ada,c++
Thread model: posix
gcc version 4.3.0 20080114 (experimental) (GCC) 
COLLECT_GCC_OPTIONS='-c' '-v' '-gnat95' '-mtune=generic'
 /opt/gccsnap/libexec/gcc/i686-pc-linux-gnu/4.3.0/gnat1 -quiet -dumpbase pkg001u.adb -gnat95 -mtune=generic pkg001u.adb -o /tmp/ccmBlEVU.s
+===========================GNAT BUG DETECTED==============================+
| 4.3.0 20080114 (experimental) (i686-pc-linux-gnu) Storage_Error heap exhausted|
| Error detected at pkg001u.adb:296:35                                     |
| Please submit a bug report; see http://gcc.gnu.org/bugs.html.            |
| Use a subject line meaningful to you and us to track the bug.            |
| Include the entire contents of this bug box in the report.               |
| Include the exact gcc or gnatmake command that you entered.              |
| Also include sources listed below in gnatchop format                     |
| (concatenated together with no headers between files).                   |
+==========================================================================+

Please include these source files with error report
Note that list may not be accurate in some cases,
so please double check that the problem can still
be reproduced with the set of files listed.

pkg001u.adb
pkg001u.ads
p_portable.ads
corba.ads
pkg001z.ads
pkg0013.ads
pkg000x.ads
pkg001o.ads
pkg000q.ads
pkg000v.ads
pkg000y.ads
pkg001l.ads
pkg0025.ads
pkg001e.ads
pkg000w.ads
pkg001r.ads
pkg000u.ads
pkg002n.ads
pkg002f.ads
pkg0020.ads
pkg0019.ads
pkg001w.ads
pkg000t.ads
pkg001v.ads
pkg001m.ads
pkg000a.ads
pkg000d.ads
pkg000c.ads
pkg0007.ads
pkg0009.ads
pkg000k.ads
pkg0001.ads
pkg000g.ads
pkg000b.ads
pkg000l.ads
pkg000f.ads
pkg002g.ads
pkg0006.ads
pkg001k.ads
pkg0000.ads
pkg0005.ads
pkg002o.ads
pkg0028.ads
pkg0029.ads
pkg000h.ads
pkg002p.ads
pkg000j.ads
pkg000i.ads
pkg0027.ads
pkg0002.ads
pkg0022.ads
pkg0023.ads
pkg0012.ads
pkg0026.ads
pkg000r.ads
pkg000s.ads
c_types.ads
pkg000m.ads
pkg001a.ads
pkg0003.ads
pkg001n.ads
pkg0021.ads
pkg001t.ads
pkg001p.ads
pkg001c.ads
pkg0004.ads
pkg0016.ads
pkg001s.ads
pkg001x.ads
pkg001z-o_protokoll.ads
pkg0018.ads
pkg000z.ads
pkg001j.ads
pkg0011.ads
pkg001d.ads
pkg002m.ads
pkg002e.ads
pkg002d.ads
pkg000o.ads
pkg001b.ads
pkg000p.ads
pkg001f.ads
pkg0015.ads
pkg000n.ads
pkg0014.ads
pkg0017.ads
pkg0024.ads

compilation abandoned

(Tried limiting the VIRT to 1.5 gig, same result.)
Comment 1 Oliver Kellogg 2008-01-21 07:30:54 UTC
Created attachment 14985 [details]
source code for producing the bug

I tried this with gcc-4.1, 4.2, and 4.3.

If I remove the aggregate assignments such as

   Fumessage.O000X := (Pkg000W.E08EY, Aktergsatz);

from pkg001u.adb then compilation proceeds at normal memory consumption.
Comment 2 Oliver Kellogg 2008-05-22 11:23:53 UTC
Still happens with 4.4.0 20080522.
Please advise if there is any further info that I could provide
to help track the problem down.
Comment 3 Oliver Kellogg 2008-05-25 10:38:52 UTC
Does not happen with -gnatc (syntax and semantics check only.)
Comment 4 Richard Biener 2008-05-25 12:03:24 UTC
Does enabling optimization (-O) fix the problem?  My guess is that the
gimplification of the aggregate assignments creates lots of overhead, but that
needs to be investigated by Ada people - stats with a compiler configured with
--enable-gather-detailed-mem-stats would also be useful.
Comment 5 Oliver Kellogg 2008-05-25 13:31:57 UTC
Created attachment 15679 [details]
statistics output from gnat1 on pkg001u.adb without aggregate assignments
Comment 6 Oliver Kellogg 2008-05-25 13:38:19 UTC
Created attachment 15680 [details]
statistics output from gnat1 on pkg001u.adb with one assignment

Here, I enabled the assignment in line 377,

   Tramessage.O000X := (Pkg000W.E08EY, Aktergsatz);

and invoked gnat1 directly as follows:

/opt/gccsnap/libexec/gcc/i686-pc-linux-gnu/4.4.0/gnat1 -gnat95 -mtune=generic \
   pkg001u.adb

(Notice the increase in 'expand'. Is that within expected limits?)
Comment 7 Richard Biener 2008-05-25 13:48:06 UTC
Well, this assignment seems to be _very_ expensive both in terms of parsing time
and size of the IL to expand.  It certainly looks unreasonable.
Comment 8 Oliver Kellogg 2008-05-25 15:42:15 UTC
(in reply to comment #4)
> Does enabling optimization (-O) fix the problem?

No, does not change the behavior (other than taking even longer)

> [...] stats with a compiler configured with
> --enable-gather-detailed-mem-stats would also be useful.

Coming up.
Comment 9 Oliver Kellogg 2008-05-25 18:12:18 UTC
Created attachment 15681 [details]
gnat1 (trunk r135848) output from -fmem-report, no aggregate assignments
Comment 10 Oliver Kellogg 2008-05-25 18:17:21 UTC
Created attachment 15682 [details]
same as above but with assignments in pkg001u.adb lines 296 and 377 enabled
Comment 11 Oliver Kellogg 2008-05-25 18:43:13 UTC
Created attachment 15683 [details]
att15682 was incorrect, two assignments already exhaust the memory. memreport for _one_ assignmt.
Comment 12 Richard Biener 2008-05-25 18:56:20 UTC
ada/utils2.c:1774 (build_simple_component_ref)    111547200:71.1%

clearly a frontend issue.
Comment 13 Oliver Kellogg 2008-06-01 18:10:21 UTC
Created attachment 15708 [details]
instrumentation of build_simple_component_ref (for analysis only)

I put in some putchar calls of different letters to see where the code is going.
The code formatting looks bad but maintains the original line numbers.
Comment 14 Oliver Kellogg 2008-06-01 18:28:22 UTC
Created attachment 15709 [details]
console output from gnat1 with the above instrumentation

The following pattern occurs extremely often:

   'C'  (DECL_CONTEXT (field) != record_type), line 1716
   'D'  (!new_field), line 1735
   'J'  (!field), line 1753, return NULL_TREE
 'H'  retval from call at line 1744 was NULL
 'E'  (DECL_INTERNAL_P (new_field)) on next iteration of loop at line 1736
   'M'  now in new call to build_simple_component_ref
   'P'  return fold (ref);   at line 1800
 'F'  just before next call to build_simple_component_ref at line 1744

and back up to 'C'.

(I am still trying to understand what is actually happening.)
Comment 15 Richard Biener 2008-06-01 18:56:45 UTC
You can look at the original IL generated, I guess the assignments simply contain
millions of element assignments (-fdump-tree-original).

??? tree nodes created

Kind                   Nodes      Bytes
...
refs                 2903482  115769372

some Ada guy needs to look into this.
Comment 16 Oliver Kellogg 2008-06-02 19:16:11 UTC
Created attachment 15715 [details]
output from -fdump-tree-original of gnat1 compiling pkg001u.adb

(in reply to comment #15)
> You can look at the original IL generated, I guess the assignments simply
> contain
> millions of element assignments (-fdump-tree-original).

Ah, thanks.
I believe the code at line 29417 represents the problematic assignment
(pkg001u.adb line 296), could somebody check?

Some of this stuff looks strange to the layman's eye, for example
around line 29402:

                    <<< Unknown tree: loop_stmt
Comment 17 Oliver Kellogg 2008-08-21 05:41:23 UTC
Created attachment 16118 [details]
regenerated statistics: trunk r139367 gnat1-gnat95 -fmem-report -fdump-tree-all pkg001u.adb
Comment 18 Eric Botcazou 2011-05-12 05:44:25 UTC
Too many layers of variant parts with too many components in them.