[Bug rtl-optimization/93658] [9/10 Regression] infinite loop building ghostscript and icu with -O3 on powerpc64le-linux-gnu

[bergner at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93658

Peter Bergner <bergner at gcc dot gnu.org> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |ASSIGNED
           Assignee|unassigned at gcc dot gnu.org      |bergner at gcc dot gnu.org

--- Comment #10 from Peter Bergner <bergner at gcc dot gnu.org> ---
Ok, after debugging, this looks to be a bug in rs6000_legitimate_address_p().
At the beginning of LRA, we have the following insn:

(insn 520 67 71 5 (set (mem:V16QI (and:DI (reg/f:DI 110 sfp)
                (const_int -16 [0xfffffffffffffff0])) [0 MEM <vector(16) char>
[(char *)&a + 16B]+0 S16 A128])
        (reg:V16QI 303 [ vect__2.13 ])) "bug.i":10:10 1103 {vsx_movv16qi_64bit}
     (expr_list:REG_DEAD (reg:V16QI 303 [ vect__2.13 ])
        (nil)))

lra_eliminate() then converts that to:

(insn 520 67 71 5 (set (mem:V16QI (and:DI (plus:DI (reg/f:DI 110 sfp)
                    (const_int 112 [0x70]))
                (const_int -16 [0xfffffffffffffff0])) [0 MEM <vector(16) char>
[(char *)&a + 16B]+0 S16 A128])
        (reg:V16QI 303 [ vect__2.13 ])) "bug.i":10:10 1103 {vsx_movv16qi_64bit}
     (expr_list:REG_DEAD (reg:V16QI 303 [ vect__2.13 ])
        (nil)))

which isn't valid and needs fixing...which is fine.  process_address() then
gives us:

(insn 520 67 71 5 (set (mem:V16QI (and:DI (reg:DI 633)
                (const_int -16 [0xfffffffffffffff0])) [0 MEM <vector(16) char>
[(char *)&a + 16B]+0 S16 A128])
        (reg:V16QI 303 [ vect__2.13 ])) "bug.i":10:10 1103 {vsx_movv16qi_64bit}
     (expr_list:REG_DEAD (reg:V16QI 303 [ vect__2.13 ])
        (nil)))

...with pseudo 633 being set to plus:DI (reg/f:DI 110 sfp) (const_int 112
[0x70])
which is also good.  The bug comes when we now pass insn 520 to
rs6000_legitimate_address_p() and we hit the following code:

  /* If this is an unaligned stvx/ldvx type address, discard the outer AND.  */
  if (VECTOR_MEM_ALTIVEC_P (mode)
      && GET_CODE (x) == AND
      && CONST_INT_P (XEXP (x, 1))
      && INTVAL (XEXP (x, 1)) == -16)
    x = XEXP (x, 0);

Our mode is V16QI, so we should execute this code, but we don't.  The problem
is that VECTOR_MEM_ALTIVEC_P(mode) checks for "rs6000_vector_unit[(MODE)] ==
VECTOR_ALTIVEC" and V16QI mode returns VECTOR_VSX, so we fail to fall into the
code above and end up telling LRA this isn't a valid address.  LRA then
attempts to spill this again and again and...   

The following patch fixes the bug for me and I'm regtesting it now:

--- a/gcc/config/rs6000/rs6000.c
+++ b/gcc/config/rs6000/rs6000.c
@@ -8808,7 +8808,7 @@ rs6000_legitimate_address_p (machine_mode mode, rtx x,
bool reg_ok_strict)
   bool quad_offset_p = mode_supports_dq_form (mode);

   /* If this is an unaligned stvx/ldvx type address, discard the outer AND. 
*/
-  if (VECTOR_MEM_ALTIVEC_P (mode)
+  if (VECTOR_MEM_ALTIVEC_OR_VSX_P (mode)
       && GET_CODE (x) == AND
       && CONST_INT_P (XEXP (x, 1))
       && INTVAL (XEXP (x, 1)) == -16)