Hello, Bootstrap of gcc 13.1.0 fails during compilation of gimple-match.cc. The error message is "virtual memory exhausted: Cannot allocate memory", possibly from the assembler. However, virtual memory is not really exhausted. A slightly different version of (possibly) the same error occurs during stage3 compilation of gimple-match.cc (see below). Setup: 1) Install QEMU. I'm using QEMU version 7.2.1 on an HP Z2 workstation (32 GiB, eight i7 CPUs) running Debian SID. 2) Install Gentoo to a disk image or a separate disk (using whatever your usual procedure is for installing Gentoo). I used the latest Gentoo stage3 for m68k for the first procedure below, and a fully installed Gentoo for the second procedure. For swap space, I tried many options, ending up with two 512 MiB swap partitions. Regardless of how much swap space is used, compilation of gimple-match.cc will fail with the same error. To generate the error in stage2 gcc bootstrap, the following steps can be used: 1) Create a QEMU virtual machine (VM) configured as q800 (e.g. "-M q800 -m 1004M") or as virt (e.g. "-M virt -m 3399672K"). Using q800, you can use the latest default Debian kernel for m68k; otherwise, you'll need a virt kernel. They will have the same error. 2) It's not necessary to configure the network, users, etc. in the Gentoo QEMU VM (instead use the chroot environment). After installation of the Gentoo stage3 as per the Gentoo installation instructions, you can run "emerge-webrsync" then "emerge gcc" from the chroot. Compilation will fail in gcc stage2. To generate the error in a manual gcc compilation, the following steps can be used: 1) If you have a completed installation of Gentoo or Debian, you can download the latest gcc. I used Gentoo's configure options (but not their patches): "../gcc-13.1.0/configure --host=m68k-unknown-linux-gnu --build=m68k-unknown-linux-gnu --prefix=/usr --bindir=/usr/m68k-unknown-linux-gnu/gcc-bin/13 --includedir=/usr/lib/gcc/m68k-unknown-linux-gnu/13/include --datadir=/usr/share/gcc-data/m68k-unknown-linux-gnu/13 --mandir=/usr/share/gcc-data/m68k-unknown-linux-gnu/13/man --infodir=/usr/share/gcc-data/m68k-unknown-linux-gnu/13/info --with-gxx-include-dir=/usr/lib/gcc/m68k-unknown-linux-gnu/13/include/g++-v13 --with-python-dir=/share/gcc-data/m68k-unknown-linux-gnu/13/python --enable-languages=c,c++,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --disable-nls --disable-libunwind-exceptions --with-gcc-major-version-only --disable-esp --enable-libstdcxx-time --enable-lto --disable-libstdcxx-pch --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --disable-multilib --disable-fixed-point --enable-libgomp --disable-libssp --disable-libada --disable-cet --disable-systemtap --disable-valgrind-annotations --disable-vtable-verify --disable-libvtv --without-zstd --without-isl --disable-libsanitizer --enable-default-pie --enable-default-ssp" followed by "make bootstrap". 2) The compilation should eventually fail in stage3 gcc bootstrap while compiling gimple-match.cc. I'll be happy to provide any additional details or clarifications that you may require. Thank you for your help. -Stan Johnson
I wonder if splitting up gimple-match that happened on the trunk fixes this issue.
(In reply to Andrew Pinski from comment #1) > I wonder if splitting up gimple-match that happened on the trunk fixes this > issue. I guess the issue is just "compiling giant TU has exhausted the memory" so it should be fixed now. But how to handle it for stable branches? :(
assign more virtual memory? the other option is to figure out what exactly uses so much memory in a native m68k compiler ... if stage2 is fine then using -g0 for the object might help. x86_64 native needs 1.3GB and using -O1 doesn't make a difference there but -g1 shrinks memory requirements to 750MB (similar to using -g -fno-var-tracking).
Adding more swap space doesn't help. The latest Gentoo update attempted to merge gcc-13.1.1_p20230520, and it failed while compiling gimple-match.cc in stage2. The error message is "virtual memory exhausted: Cannot allocate memory". Here are some memory details just before the failure: ./work/build/./prev-gcc/cc1plus VSZ=920260 RSS=632200 /usr/m68k-unknown-linux-gnu/bin/as VSZ=265260 RSS=258608 There is still swap space available: # cat /proc/swaps Filename Type Size Used Priority /dev/sda7 partition 524284 225008 -2 /dev/sda8 partition 524284 0 -3
Thanks for reporting this upstream, Stan. We've been hitting this for a while in our QEMU builds in Gentoo but we hadn't got to the point where we'd figured out who to report it to.
Thanks, let me know if you need me to check anything here. The problem has existed since at least gcc-12. Initially I thought the QEMU q800 VM had run out of memory, but switching to a virt VM with ~3 GB memory and 1 GiB swap also failed. Further checking showed that the virt VM was not actually out of memory (unless I wasn't able to monitor memory changes quickly enough looking at /proc/swaps and /proc/meminfo every minute).
How much virtual memory does the m68k host have? Is it possible to extend that to 2GB? ISTR that linux on 32bit platforms had a 1GB userland virtual memory setup at some point (and maybe for some platforms). In theory it's possible to backport the split of the gimple-match.cc TUs. Can you try current mainline to see if that's enough to pass native bootstrap in qemu?
> How much virtual memory does the m68k host have? Swap space in the m68k virt VM is configurable; I'm using 1 GiB in two 512 MiB partitions. I noticed that compilation goes further with two 512 MiB partitions that with one 1 GiB partition, even though the second 512 MiB swap partition isn't used (see above). And I know I tested 2 GiB swap at one point, and that also failed. The virt VM is configured with 3399672K memory. While observing the failure, it appears to happen when the assembler "as" becomes more active (so probably when cc1plus is about done). Assembler memory increases quickly, and swap space is not really exhausted, despite the error message. > Can you try current mainline to see if that's enough to pass native > bootstrap in qemu? Native (manual) bootstrap of gcc-13.1 in QEMU (virt and q800) failed in Gentoo, but the failure was slightly different (it failed in stage 3 instead of stage 2). And I used Gentoo's configure options, as documented above. I could try no configure options, followed by "make bootstrap"; would that be helpful? If yes, should I use Gentoo or Debian SID?
On Tue, 23 May 2023, userm57 at yahoo dot com wrote: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 > > --- Comment #8 from Stan Johnson <userm57 at yahoo dot com> --- > > How much virtual memory does the m68k host have? > > Swap space in the m68k virt VM is configurable; I'm using 1 GiB in two 512 MiB > partitions. I noticed that compilation goes further with two 512 MiB partitions > that with one 1 GiB partition, even though the second 512 MiB swap partition > isn't used (see above). And I know I tested 2 GiB swap at one point, and that > also failed. The virt VM is configured with 3399672K memory. The question is how much virtual memory is exposed to a user process, that is - how large is the address space? > While observing the failure, it appears to happen when the assembler "as" > becomes more active (so probably when cc1plus is about done). Assembler memory > increases quickly, and swap space is not really exhausted, despite the error > message. That's odd, I never noticed the assembler taking much memory. > > Can you try current mainline to see if that's enough to pass native > > bootstrap in qemu? > > Native (manual) bootstrap of gcc-13.1 in QEMU (virt and q800) failed in Gentoo, > but the failure was slightly different (it failed in stage 3 instead of stage > 2). And I used Gentoo's configure options, as documented above. I could try no > configure options, followed by "make bootstrap"; would that be helpful? If yes, > should I use Gentoo or Debian SID? Note mainline would be gcc 14.0, you can probably download a recent snapshot.
> The question is how much virtual memory is exposed to a user > process, that is - how large is the address space? I'm not sure, but I see this: $ prlimit RESOURCE DESCRIPTION SOFT HARD UNITS AS address space limit unlimited unlimited bytes CORE max core file size 0 unlimited bytes CPU CPU time unlimited unlimited seconds DATA max data size unlimited unlimited bytes FSIZE max file size unlimited unlimited bytes LOCKS max number of file locks held unlimited unlimited locks MEMLOCK max locked-in-memory address space 8388608 8388608 bytes MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes NICE max nice prio allowed to raise 0 0 NOFILE max number of open files 1024 4096 files NPROC max number of processes 26236 26236 processes RSS max resident set size unlimited unlimited bytes RTPRIO max real-time priority 0 0 RTTIME timeout for real-time tasks unlimited unlimited microsecs SIGPENDING max number of pending signals 26236 26236 signals STACK max stack size 8388608 unlimited bytes As long as those are also the limits for portage, it should be ok. > Note mainline would be gcc 14.0, you can probably download a recent > snapshot. ok, I've downloaded the latest gcc snapshot using git. I'll try a manual bootstrap of that. Do you recommend that I use QEMU m68k (virt) running Debian SID or Gentoo, or doesn't it matter?
> Am 23.05.2023 um 19:28 schrieb userm57 at yahoo dot com <gcc-bugzilla@gcc.gnu.org>: > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 > > --- Comment #10 from Stan Johnson <userm57 at yahoo dot com> --- >> The question is how much virtual memory is exposed to a user >> process, that is - how large is the address space? > > I'm not sure, but I see this: > > $ prlimit > RESOURCE DESCRIPTION SOFT HARD UNITS > AS address space limit unlimited unlimited bytes That’s indeed unhelpful. Cat /proc/maps of some executable might help > CORE max core file size 0 unlimited bytes > CPU CPU time unlimited unlimited seconds > DATA max data size unlimited unlimited bytes > FSIZE max file size unlimited unlimited bytes > LOCKS max number of file locks held unlimited unlimited locks > MEMLOCK max locked-in-memory address space 8388608 8388608 bytes > MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes > NICE max nice prio allowed to raise 0 0 > NOFILE max number of open files 1024 4096 files > NPROC max number of processes 26236 26236 processes > RSS max resident set size unlimited unlimited bytes > RTPRIO max real-time priority 0 0 > RTTIME timeout for real-time tasks unlimited unlimited microsecs > SIGPENDING max number of pending signals 26236 26236 signals > STACK max stack size 8388608 unlimited bytes > > As long as those are also the limits for portage, it should be ok. > >> Note mainline would be gcc 14.0, you can probably download a recent >> snapshot. > > ok, I've downloaded the latest gcc snapshot using git. I'll try a manual > bootstrap of that. Do you recommend that I use QEMU m68k (virt) running Debian > SID or Gentoo, or doesn't it matter? I don’t think that matters.
> That’s indeed unhelpful. Cat /proc/maps of some executable might help Sorry, I don't know how to do that. I don't see /proc/maps. Regarding downloading mainline gcc-14, it's not at ftp.gnu.org. I tried the following, but I must be using the wrong repository. Please advise on where I should find gcc-14. $ git clone git://gcc.gnu.org/git/gcc.git $ cd gcc $ git reset --hard HEAD is now at 0d2ab259bc5 Improve cost computation for single-bit bit insertions. $ git checkout releases/gcc-14 error: pathspec 'releases/gcc-14' did not match any file(s) known to git $ git checkout releases/gcc-13 Already on 'releases/gcc-13' Your branch is up to date with 'origin/releases/gcc-13'. $ git checkout mainline/gcc-14 error: pathspec 'mainline/gcc-14' did not match any file(s) known to git
> Am 23.05.2023 um 19:44 schrieb userm57 at yahoo dot com <gcc-bugzilla@gcc.gnu.org>: > > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 > > --- Comment #12 from Stan Johnson <userm57 at yahoo dot com> --- >> That’s indeed unhelpful. Cat /proc/maps of some executable might help > > Sorry, I don't know how to do that. I don't see /proc/maps. Try /proc/1/maps It’s a file in the per process directories > Regarding downloading mainline gcc-14, it's not at ftp.gnu.org. I tried the > following, but I must be using the wrong repository. Please advise on where I > should find gcc-14. > > $ git clone git://gcc.gnu.org/git/gcc.git > $ cd gcc > $ git reset --hard > HEAD is now at 0d2ab259bc5 Improve cost computation for single-bit bit > insertions. > $ git checkout releases/gcc-14 > error: pathspec 'releases/gcc-14' did not match any file(s) known to git > $ git checkout releases/gcc-13 > Already on 'releases/gcc-13' > Your branch is up to date with 'origin/releases/gcc-13'. > $ git checkout mainline/gcc-14 > error: pathspec 'mainline/gcc-14' did not match any file(s) known to git Git checkout master Should do what I meant
I can try to capture the offending "cc1plus" and "as" processes just before the compilation of gimple-match.cc fails in stage2; the output will look something like this (for a different cc1plus process number 19431 that is currently running): # cat /proc/19431/maps 80000000-82027000 r-xp 00000000 08:05 1823425 /var/tmp/portage/sys-devel/gcc-13.1.1_p20230520/work/build/prev-gcc/cc1plus 82028000-8202a000 r--p 02028000 08:05 1823425 /var/tmp/portage/sys-devel/gcc-13.1.1_p20230520/work/build/prev-gcc/cc1plus 8202a000-82036000 rw-p 0202a000 08:05 1823425 /var/tmp/portage/sys-devel/gcc-13.1.1_p20230520/work/build/prev-gcc/cc1plus 82036000-82070000 rwxp 00000000 00:00 0 [heap] 82070000-82601000 rwxp 00000000 00:00 0 [heap] c0000000-c0020000 r-xp 00000000 08:05 1682956 /lib/ld.so.1 c0020000-c0022000 r--p 00020000 08:05 1682956 /lib/ld.so.1 c0022000-c0023000 rw-p 00022000 08:05 1682956 /lib/ld.so.1 c0023000-c0026000 rw-p 00000000 00:00 0 c0028000-c0044000 r-xp 00000000 08:05 1515049 /usr/lib/libmpc.so.3.3.1 c0044000-c0045000 ---p 0001c000 08:05 1515049 /usr/lib/libmpc.so.3.3.1 c0045000-c0046000 r--p 0001b000 08:05 1515049 /usr/lib/libmpc.so.3.3.1 c0046000-c0047000 rw-p 0001c000 08:05 1515049 /usr/lib/libmpc.so.3.3.1 c0048000-c00b3000 r-xp 00000000 08:05 1672972 /usr/lib/libmpfr.so.6.2.0 c00b3000-c00b5000 ---p 0006b000 08:05 1672972 /usr/lib/libmpfr.so.6.2.0 c00b5000-c00b6000 r--p 0006b000 08:05 1672972 /usr/lib/libmpfr.so.6.2.0 c00b6000-c00b7000 rw-p 0006c000 08:05 1672972 /usr/lib/libmpfr.so.6.2.0 c00b8000-c0113000 r-xp 00000000 08:05 1515186 /usr/lib/libgmp.so.10.4.1 c0113000-c0115000 ---p 0005b000 08:05 1515186 /usr/lib/libgmp.so.10.4.1 c0115000-c0116000 r--p 0005b000 08:05 1515186 /usr/lib/libgmp.so.10.4.1 c0116000-c0117000 rw-p 0005c000 08:05 1515186 /usr/lib/libgmp.so.10.4.1 c0118000-c012a000 r-xp 00000000 08:05 411479 /lib/libz.so.1.2.13 c012a000-c012b000 ---p 00012000 08:05 411479 /lib/libz.so.1.2.13 c012b000-c012c000 r--p 00011000 08:05 411479 /lib/libz.so.1.2.13 c012c000-c012d000 rw-p 00012000 08:05 411479 /lib/libz.so.1.2.13 c012e000-c0172000 r-xp 00000000 08:05 1682625 /lib/libm.so.6 c0172000-c0173000 ---p 00044000 08:05 1682625 /lib/libm.so.6 c0173000-c0174000 r--p 00043000 08:05 1682625 /lib/libm.so.6 c0174000-c0176000 rw-p 00044000 08:05 1682625 /lib/libm.so.6 c0176000-c02df000 r-xp 00000000 08:05 1682906 /lib/libc.so.6 c02df000-c02e0000 ---p 00169000 08:05 1682906 /lib/libc.so.6 c02e0000-c02e2000 r--p 0016a000 08:05 1682906 /lib/libc.so.6 c02e2000-c02e6000 rw-p 0016c000 08:05 1682906 /lib/libc.so.6 c02e6000-c0ad2000 rw-p 00000000 00:00 0 c0ad5000-c25dd000 rw-p 00000000 00:00 0 ef91f000-ef948000 rw-p 00000000 00:00 0 [stack] > Git checkout master Thanks, I'll try compiling that version manually and also capture the relevant /proc output.
TASK_SIZE is 0xF0000000UL on m68k.
(In reply to Andreas Schwab from comment #15) > TASK_SIZE is 0xF0000000UL on m68k. That would mean ~3.75GB virtual address space is available. The cited /proc/maps though looks like the lower half isn't available? It would be probably interesting to strace the process that will ultimatively run out of virtual memory to see how glibc places allocations. A simple test program repeatedly creating anonymous mmaps of a larger size (2MB?) might also get an upper bound.
The linker just uses TEXT_START_ADDR=0x80000000, but mmap can use any address.
$ git clone git://gcc.gnu.org/git/gcc.git $ cd gcc $ git checkout master I'm testing a manual bootstrap of "gcc version 14.0.0 20230524 (experimental) (GCC)" now, accessed via git as shown above. It will still take about 24 more hours for the bootstrap to finish (I'll send an update if it fails), but with gimple-match.cc (and generic-match.cc, which was not affected in my tests) split up, it looks like it will finish ok (it's currently in about the middle of stage 2 and has successfully compiled all the gimple-match-n.cc files). Note that Gentoo's emerge of gcc-13 behaves a little differently than a manual bootstrap. I don't know why, since I think I'm using Gentoo's ./configure options in the manual bootstrap, but in Gentoo's emerge of gcc, they seem to run cc1plus and "as" simultaneously for each compilation, perhaps aggravating the memory issue for gimple-match.cc (or maybe not, since the problem is virtual memory exhausted, not swap space exhausted). Anyway, it looks like the solution was already close. Does anyone know whether the change will be backported to gcc-12 or gcc-13 available from ftp.gnu.org/pub/gnu/gcc? Thanks to all of the GNU developers who continue to make modern tools available for use on old hardware!
On Wed, 24 May 2023, userm57 at yahoo dot com wrote: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 > > --- Comment #18 from Stan Johnson <userm57 at yahoo dot com> --- > $ git clone git://gcc.gnu.org/git/gcc.git > $ cd gcc > $ git checkout master > > I'm testing a manual bootstrap of "gcc version 14.0.0 20230524 (experimental) > (GCC)" now, accessed via git as shown above. > > It will still take about 24 more hours for the bootstrap to finish (I'll send > an update if it fails), but with gimple-match.cc (and generic-match.cc, which > was not affected in my tests) split up, it looks like it will finish ok (it's > currently in about the middle of stage 2 and has successfully compiled all the > gimple-match-n.cc files). > > Note that Gentoo's emerge of gcc-13 behaves a little differently than a manual > bootstrap. I don't know why, since I think I'm using Gentoo's ./configure > options in the manual bootstrap, but in Gentoo's emerge of gcc, they seem to > run cc1plus and "as" simultaneously for each compilation, perhaps aggravating > the memory issue for gimple-match.cc (or maybe not, since the problem is > virtual memory exhausted, not swap space exhausted). Possibly -pipe is used somehow. > Anyway, it looks like the solution was already close. Does anyone know whether > the change will be backported to gcc-12 or gcc-13 available from > ftp.gnu.org/pub/gnu/gcc? That wasn't planned. It's very unlikely to happen for gcc 12, it should be possible for gcc 13 I think.
When I looked at it, I think I got it to apply to 13 with no hassle and it seemed to work okay, but I didn't test it that hard. It's a considerable win so even if not backported upstream, if you think it's safe enough, let us know if it's ok for us to do it downstream in Gentoo. (That could also serve as a form of testing to do it upstream on releases/gcc-13 if desired.)
On Thu, 25 May 2023, sjames at gcc dot gnu.org wrote: > https://gcc.gnu.org/bugzilla/show_bug.cgi?id=109927 > > --- Comment #20 from Sam James <sjames at gcc dot gnu.org> --- > When I looked at it, I think I got it to apply to 13 with no hassle and it > seemed to work okay, but I didn't test it that hard. > > It's a considerable win so even if not backported upstream, if you think it's > safe enough, let us know if it's ok for us to do it downstream in Gentoo. > > (That could also serve as a form of testing to do it upstream on > releases/gcc-13 if desired.) yes, it's safe (unless there are not yet noticed portability issues with the Makefile setup)
Thanks for your help. We've had this in production since 26th May and no complaints (but a lot of happy customers who couldn't build 12 or earlier 13 versions): https://gitweb.gentoo.org/proj/gcc-patches.git/commit/?id=7a6b2d23ec02e75475a6123254ccd44d73827f39. Now that 13.2 is done, could this be pushed to the branch for 13.3?
Just to summarize we are talking about the six commits r14-496-g580cda3c2799b1 r14-497-ge487fcc0f7466e r14-498-gc0ce29bc1ce329 r14-499-g27fcf994c5515e r14-500-g703417a030b3d8 r14-501-g0a85544e1aaeca there were related followups in r14-535-g319aef8d2fde32, r14-570-g31c70a7daa3687, r14-599-g2ed6dd975d1da8, r14-600-g425198bb5d2e30 and r14-601-g153eafaa0b7449
I think we can call this fixed via the split improvements. It's in 14 but it's also in 13 downstream in Gentoo where Stan reported it from.