When compiling gcc-4.1.1, there is at least a serious slowdown, and on many machines a failure, when insn-attrtab is compiled. There are several threads on the Gentoo forums about this, some involving machines with 512M of memory. This file seems to be automatically generated, and has become so large that one user graphically describes it as a "swapfest". Compiling on a machine with the specified Gentoo minimum of 64M is not practical. Is it possible to split this file?
> Compiling on a machine with the > specified Gentoo minimum of 64M is not practical. HEHEHEHEHEHEHEHE. 64MB is very very small ammount of memory really. Even the PS3 has 256MB. Oh and compiling GCC should not have a memory requirement so gentoo is doing something wrong.
the "minimum" is a suggestion in our install docs ... it's like the "minimum" you see on the side of boxed software in the store the 64M has nothing to do with GCC
I think the pertinent part of this "bug" is: > Is it possible to split this file? It took 6-8 hours to compile this file alone on my xbox, which only has 64m of memory but should be considered a sufficiently modern machine. My understanding is that this is a generated file, containing information about lots of different architectures. Might it be possible to generate a file per architecture, or per group of architectures? This is not a new problem, this file is what made me give up on Linux for the Dreamcast (Which might not be considered a sufficiently modern machine.)
The problem is not only confined to "sufficiently modern machines". Some of us have a conservationist sort of outlook, and are happy to find a use for an older machine as a router, firewall, Kerberos server or whatever. Linux supports this sort of thing well - a floppy firewall with no hard drive will run just fine on an 8M machine, and a 486/33 will keep up with all that an ordinary ADSL connection can transmit. With insn-attrtab excluded, gcc-4.1.1 compiles quickly and well on 52M, and quite possibly on smaller machines, though I have no very recent experience to prove it. It seems a pity that a single source file, and an automatically-generated one at that, should essentially block an excellent compiler from a lot of smaller machines, new and old, and cause enough of a slowdown even on a 512M machine to arouse concern. If there is a rational reason for this file to be huge, well and good, but if it is readily split that would seem like a worthwhile thing to do.
I have experienced the same problem. I am compiling gcc on a machine with 32 mb ram. Even with over 2 Gb swap it will work for hours and then finally crash with an out of memory exception. I have worked around the problem by compiling gcc on another machine, and successfully resumed the build. (no word if it worked yet). Are there any patches that fix/ alleviate this problem?
As far as I can tell (from running cc1 in a debugger), the problem is not so much the size of the file, but that it contains two large functions and GCC leaks memory. After compiling the first large function, the process size is 886 megs (basic-block reordering alone being responsible for about 200 megs), probably littered with pieces of garbage. That'll cause all following allocations to become scattered thoughout memory and the swapfest begins if you don't have lots of memory. 256 megs is not enough. The RTL passes in particular seem to be affected and several df_* functions take ages to run. I'm working on a patch to split insn-attrtab.c into more files.
Created attachment 14420 [details] Reenable alloca() on non-GCC compilers The memory fragmentation problem is to be caused by libiberty which disables alloca() if you bootstrap with a non-GCC compiler[1] (which I do). Instead you get a malloc() based replacment. The df_* functions use alloca() and are called a lot. With this patch on top of the next one, process size tops at around 185 MB compared to the 900+ MB it reached before, at which point I gave up with 256 MB of RAM. Just wondering, which compiler are you using to bootstrap GCC? [1] http://gcc.gnu.org/ml/gcc-patches/2001-03/msg00194.html
Created attachment 14421 [details] split insn-attrtab.c into three files Here's the patch to split insn-attrtab.c into smaller pieces. The result: $ wc -l insn-*tab.c 14566 insn-attrtab.c 44454 insn-dfatab.c 36815 insn-latencytab.c 95835 total Please tell me if it makes a difference. (It is necessary to run autoheader and autoconf in the gcc directory after applying the patches.)
I just tried with only the alloca patch, and despite the unsplit insn-attrtab.c file, process size tops at just 205 MB. It looks like GCC 4.3.0 is in a much better shape than GCC 4.1.1, so I'm letting go of the bug. Just to clarify, the process sizes where taken when the stage1 compiler was compiling the stage2 compiler.
What happened with the alloca patch? Looks like it was never committed.
I can confirm this is still a problem on 2.5.3. Compiling on a system with 256MB memory (hired, remote, virtualized) is not possible without an extra 1GB of memory and almost nothing else running (the memory usage peaked at full RAM and 522MB swap used, with gcc using everything but 60MB).
(In reply to comment #11) > I can confirm this is still a problem on 2.5.3. Compiling on a system with Gašper, nothing will change unless people like you that care about such systems contribute patches to fix this situation: http://gcc.gnu.org/contribute.html
Author: steven Date: Fri May 4 20:04:47 2012 New Revision: 187181 URL: http://gcc.gnu.org/viewcvs?root=gcc&view=rev&rev=187181 Log: PR other/29442 * read-md.c (fprint_md_ptr_loc, fprint_c_condition): New functions. (print_md_ptr_loc, print_c_condition): Use them. * read-md.h (fprint_md_ptr_loc, fprint_c_condition): New prototypes. * genattrtab.c (attr_file_name, dfa_file_name, latency_file_name, attr_file, dfa_file, latency_file): New global variables. (write_attr_valueq, write_attr_set, write_attr_case, write_attr_value, write_upcase, write_indent, write_length_unit_log, write_test_expr, write_attr_get, write_insn_cases, write_eligible_delay, write_const_num_delay_slots): Accept FILE pointer and toss it around. Update all callers. (write_header, open_outfile, handle_arg): New funcions. (make_automaton_attrs): Write prototypes as extern to the output files. (main): Use init_rtx_reader_args_cb with handle_arg to take 3 file names from the command line. Open the output files and write out internal functions for DFA functions to dfa_file_name, insn latency functions to latency_file_name, and everything else to attr_file. * Makefile.in (OBJS): Add insn-dfatab.o and insn-latencytab.o. (BACKEND): Build libbackend first. (MOSTLYCLEANFILES): Add insn-dfatab.c and insn-latencytab.c. (.PRECIOUS): Likewise. (insn-dfatab.o): New rule. (insn-latencytab.o): New rule. (simple_rtl_generated_c): Do not include insn-attrtab.c. (s-attrtab): New rule. Modified: trunk/gcc/ChangeLog trunk/gcc/Makefile.in trunk/gcc/genattrtab.c trunk/gcc/read-md.c trunk/gcc/read-md.h
The patch of comment #13 and the observation of comment #8 are Good Enough for me to consider this problem fixed.
While looking for ways to speed up genattrtab itself, I found this patch, which doesn't speed up genattrtab, but would make the insn-*tab.c files smaller: http://gcc.gnu.org/ml/gcc-patches/2003-09/msg00316.html Micha, this appears to have been dropped. Any reason why you never asked for an OK to commit this patch?
I must say I don't like the calling of potentially expensive get_attr_* upfront. Can't we instead introduce inlines, get_attr_*_cached (or macros), which would take another argument by reference, initially initialized to a value don't know by genattrtab, and they would cache the attribute value in that variable. genattrtab would emit those if there is more than one call to that get_attr_* for the insn.
Seems my memory doesn't serve me well already, I've even implemented that: http://gcc.gnu.org/ml/gcc-patches/2010-06/msg01908.html
(In reply to comment #15) > While looking for ways to speed up genattrtab itself, I found this patch, which > doesn't speed up genattrtab, but would make the insn-*tab.c files smaller: > > http://gcc.gnu.org/ml/gcc-patches/2003-09/msg00316.html > > Micha, this appears to have been dropped. Any reason why you never asked for an > OK to commit this patch? I did on multiple occasions (last was http://gcc.gnu.org/ml/gcc-patches/2010-06/msg01521.html) , never got an okay. Meanwhile Jakub implemented a different mean which doesn't call the get_attr_ functions dynamically more often (which my approach potentially does), but which as result isn't nearly as effective in reducing size and compile and generation time. Some of the hunks of my patch still are useful cleanups/speedups, lemme submit them.
(In reply to comment #18) > (In reply to comment #15) > Meanwhile Jakub implemented a different mean which doesn't call the get_attr_ > functions dynamically more often (which my approach potentially does), but > which as result isn't nearly as effective in reducing size and compile and > generation time. For reducing compile time (but not size/generation), we could just use the cache vars with some magic unset value (probably the cache vars would need to be int instead of enum for that). Then if genattrtab finds out that there is no dominating call to the get_attr_* function, but still potentially for one insn the same get_attr_* might be called more than once in certain code path, then for that code path it could just use (cache_FOO != -1 ? (enum ...) cache_FOO : (cache_FOO = get_attr_FOO (...))).
(In reply to Jakub Jelinek from comment #19) > (In reply to comment #18) > > (In reply to comment #15) > > Meanwhile Jakub implemented a different mean which doesn't call the get_attr_ > > functions dynamically more often (which my approach potentially does), but > > which as result isn't nearly as effective in reducing size and compile and > > generation time. > > For reducing compile time (but not size/generation), we could just use the > cache vars with some magic unset value (probably the cache vars would need > to be int > instead of enum for that). Then if genattrtab finds out that there is no > dominating call to the get_attr_* function, but still potentially for one > insn the same get_attr_* might be called more than once in certain code > path, then > for that code path it could just use (cache_FOO != -1 ? (enum ...) cache_FOO > : (cache_FOO = get_attr_FOO (...))). If these are things that could still be done, would it make sense to reopen this bug, or open a new one for them?