Which version is your host compiler?

riscv has many many rtl patterns which definitely does not help insn-opinit size.

yeah, the split needs doing anyway, but it's really especially bad on riscv..

same version, r14-8314-g29f931e39f2

Yes, this is a known issue and it's due to our large number of patterns.  Contrary to insn-emit insn-opinit cannot be split that easily.  It would probably need a tree-like approach or similar.
I wouldn't see this as a regression in the classical sense as we just have many more patterns because of the vector extension.
Is increasing the available memory an option in the meantime or does this urgently require fixing?

The source isn't unreasonable, we should see what takes all the memory there.

Ok, I'm going to check.

My host compiler (x86_64, older trunk) uses "just" 800MB.  3.5GB looks like a runaway?  What uarch is your i586 compiler targeting?

Ah, I can reproduce - there are allocation spikes with a i586 compiler,
always during DF which oddly enough do not happen with a x86_64 compiler.

The course of action would be to reduce the testcase to a single function
(I've starting from a x86_64->riscv cross tree preprocessed source with -m32
added, so a bit of a "wrong" testcase).  Just keeping init_all_optabs
seems to reproduce it.

Reducing the number of stores to ena[] makes it eventually fit - cutting
in half seems to work for me but I still see a 1.5GB peak.

Oddly enough a checking enabled x86_64 hosted compiler shows similar bad
performance.

IIRC DF issues with many adjacent stores are not unheard of.  Possibly
on x86_64 store-merging helps to avoid this.

That said, a "fix" could be to adjust the insn-opinit generator to
emit multiple init_all_optabs functions, doing 1000 at a time.

btw, -fno-var-tracking also greatly improves compile-time (but does nothing to memory use).  Compiling with -O1 reduces memory use to 300MB even when
var-tracking is enabled.  So an option might be to force building this
generator file with -O1 (I can hardly see anything in there that would
require more).  On the DF side this replaces LIVE with LR_IN IIRC.
It's also in line with us suggesting -O1 for (large) machine generated code ...

> Is increasing the available memory an option
> in the meantime or does this urgently require fixing?

there is a buffer of 500mb, but it's already using 3.5gb.  That probably would work building without any parallelism with some Makefile changes, but building the whole compiler sequentially because of that is not a good option.

Created attachment 57209 [details]
Tentative

I tested the attached "fix".  On my machine with 13.2 host compiler it reduced the build time for insn-opinit.cc from > 4 mins to < 2 mins and the memory usage from >1G to 600ish M.  I didn't observe 3.5G before, though.

For now I just went with an arbitrary threshold of 5000 patterns and splitting into 10 functions.  After testing on x86 and aarch64 I realized that both have <3000 patterns so right now it would only split riscv's init function.

Or rather the other way, i.e. splitting into fixed-size chunks (of 1000) instead?

(In reply to Robin Dapp from comment #12)
> Created attachment 57209 [details]
> Tentative
> 
> I tested the attached "fix".  On my machine with 13.2 host compiler it
> reduced the build time for insn-opinit.cc from > 4 mins to < 2 mins and the
> memory usage from >1G to 600ish M.  I didn't observe 3.5G before, though.
> 
> For now I just went with an arbitrary threshold of 5000 patterns and
> splitting into 10 functions.  After testing on x86 and aarch64 I realized
> that both have <3000 patterns so right now it would only split riscv's init
> function.
> 
> Or rather the other way, i.e. splitting into fixed-size chunks (of 1000)
> instead?

Yeah, I'd simplify it by doing exactly that.

Ok, running tests with the adjusted version and going to post a patch afterwards.

However, during a recent run compiling insn-recog took 2G and insn-emit-7 as well as insn-emit-10 required > 1.5G each.  Looks like they could cause problems as well then?  The insn-emit files can be split into 20 instead of 10 which might help but insn-recog I haven't had a look at yet.

The master branch has been updated by Robin Dapp <rdapp@gcc.gnu.org>:

https://gcc.gnu.org/g:861997a9c7088da25ed4dc0bd339060ca063514f

commit r14-8457-g861997a9c7088da25ed4dc0bd339060ca063514f
Author: Robin Dapp <rdapp@ventanamicro.com>
Date:   Wed Jan 24 17:28:31 2024 +0100

    genopinit: Split init_all_optabs [PR113575].
    
    init_all_optabs initializes > 10000 patterns for riscv targets.  This
    leads to pathological situations in dataflow analysis (which can occur
    with many adjacent stores).
    To alleviate this this patch makes genopinit split the init_all_optabs
    function into several init_optabs_xx functions that each initialize 1000
    patterns.
    
    With this change insn-opinit.cc's compilation time is reduced from 4+
    minutes to 1:30 and memory consumption decreases from 1.2G to 630M.
    
    gcc/ChangeLog:
    
            PR other/113575
    
            * genopinit.cc (main): Split init_all_optabs into functions
            of 1000 patterns each.

Fixed on the trunk.

Forgot to change state.  Fixed on the trunk.