[Bug rtl-optimization/77844] [5/7 Regression] Compilation of simple C++ example exhaust memory

[jakub at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77844

--- Comment #8 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
For a single bb function I don't see how there could be a problem with
converging during var-tracking.
I think the problem is elsewhere, during combine.
E.g. we have:
(debug_insn 3303 3302 3304 2 (var_location:QI __x (ne:QI (ne:QI (and:DI (reg:DI
118 [ cstore_186 ])
                (const_int 262144 [0x40000]))
            (const_int 0 [0]))
        (ne:QI (and:DI (reg:DI 118 [ cstore_186 ])
                (const_int 8388608 [0x800000]))
            (const_int 0 [0])))) -1
     (nil))
and combine calls propagate_for_debug that includes this and replaces
(reg:DI 118 [ cstore_186 ])
with
(if_then_else:DI (eq (reg:CCZ 17 flags)
        (const_int 0 [0]))
    (and:DI (reg:DI 109 [ _133 ])
        (const_int -4194305 [0xffffffffffbfffff]))
    (reg:DI 320))
so it has 4 pseudo references instead of 2.
After a while it is called again replacing
(reg:DI 109 [ _133 ])
with
(if_then_else:DI (eq (reg:CCZ 17 flags)
        (const_int 0 [0]))
    (and:DI (reg:DI 104 [ _103 ])
        (const_int -2097153 [0xffffffffffdfffff]))
    (reg:DI 356))
shortly it becomes:
(debug_insn 3303 3302 3304 2 (var_location:QI __x (ne:QI (ne:QI (and:DI
(if_then_else:DI (eq (reg:CCZ 17 flags)
                        (const_int 0 [0]))
                    (and:DI (if_then_else:DI (eq (reg:CCZ 17 flags)
                                (const_int 0 [0]))
                            (and:DI (reg:DI 104 [ _103 ])
                                (const_int -2097153 [0xffffffffffdfffff]))
                            (reg:DI 356))
                        (const_int -4194305 [0xffffffffffbfffff]))
                    (ior:DI (if_then_else:DI (eq (reg:CCZ 17 flags)
                                (const_int 0 [0]))
                            (and:DI (reg:DI 104 [ _103 ])
                                (const_int -2097153 [0xffffffffffdfffff]))
                            (reg:DI 356))
                        (const_int 4194304 [0x400000])))
                (const_int 262144 [0x40000]))
            (const_int 0 [0]))
        (ne:QI (and:DI (if_then_else:DI (eq (reg:CCZ 17 flags)
                        (const_int 0 [0]))
                    (and:DI (if_then_else:DI (eq (reg:CCZ 17 flags)
                                (const_int 0 [0]))
                            (and:DI (reg:DI 104 [ _103 ])
                                (const_int -2097153 [0xffffffffffdfffff]))
                            (reg:DI 356))
                        (const_int -4194305 [0xffffffffffbfffff]))
                    (ior:DI (if_then_else:DI (eq (reg:CCZ 17 flags)
                                (const_int 0 [0]))
                            (and:DI (reg:DI 104 [ _103 ])
                                (const_int -2097153 [0xffffffffffdfffff]))
                            (reg:DI 356))
                        (const_int 4194304 [0x400000])))
                (const_int 8388608 [0x800000]))
            (const_int 0 [0])))) -1
     (nil))
and that grows further and further into a huge expression that is very
expensive to handle e.g. in nonzero_bits, etc.
So, I wonder if we don't want something in propagate_for_debug if
simplify_replace_fn_rtx returns already too complex expression to split it
apart using debug temporaries into smaller parts (or another alternative is to
reset the debug insn if it grows into a too complex expression).
In var-tracking.c we have also params that punt over certain expression
complexity (--param max-vartrack-expr-depth=12), which is kind of an example
that very complex expressions can be thrown away.
Or in cfgexpand.c avoid_deep_ter_for_debug is an example of function that
splits too complex debug expressions (over hardcoded depth of TER).