Confirmed.

In function rs6000_expand_builtin, we have one hunk to check if one bif has been enabled under the expected target flags, otherwise it will emit error but still tries to expand the call.

  if (!(e == ENB_ALWAYS
	|| (e == ENB_P5 && TARGET_POPCNTB)
	|| (e == ENB_P6 && TARGET_CMPB)
        ...

	|| (e == ENB_P10_64 && TARGET_POWER10 && TARGET_POWERPC64)
	|| (e == ENB_MMA && TARGET_MMA)))
    {
      rs6000_invalid_builtin (fcode);
      return expand_call (exp, target, ignore);
    }

I simply hacked it into:

diff --git a/gcc/config/rs6000/rs6000-builtin.cc b/gcc/config/rs6000/rs6000-builtin.cc
index 5d34c1bcfc9..f888ce80c62 100644
--- a/gcc/config/rs6000/rs6000-builtin.cc
+++ b/gcc/config/rs6000/rs6000-builtin.cc
@@ -3284,7 +3284,7 @@ htm_expand_builtin (bifdata *bifaddr, rs6000_gen_builtins fcode,
    Use the new builtin infrastructure.  */
 rtx
 rs6000_expand_builtin (tree exp, rtx target, rtx /* subtarget */,
-                       machine_mode /* mode */, int ignore)
+                       machine_mode /* mode */, int ignore ATTRIBUTE_UNUSED)
 {
   tree fndecl = TREE_OPERAND (CALL_EXPR_FN (exp), 0);
   enum rs6000_gen_builtins fcode
@@ -3394,7 +3394,7 @@ rs6000_expand_builtin (tree exp, rtx target, rtx /* subtarget */,
         || (e == ENB_MMA && TARGET_MMA)))
     {
       rs6000_invalid_builtin (fcode);
-      return expand_call (exp, target, ignore);
+      return const0_rtx;
     }

   if (bif_is_nosoft (*bifaddr)

and it's bootstrapped and no regression failures found on ppc64le p9/p10 and ppc64 p8. It's a bit surprising to me. By checking its history, it's added by r0-113861-g15ce64af29141f.  Either we miss some coverage here or it's unused any more.

I'm not sure why we still want to expand the call further, I guess we still want the bif to act like a normal function call and can link with some library definition?

And I am also curious about what will we miss if just changing it back to return const0_rtx?

Back to the failure itself, without TARGET_MMA set we don't have the optab movoo support. When it is expanding the bif, it tries to emit_move_insn (target, valreg)

(gdb) pr target
(reg:OO 117 [ _1 ])

(gdb) pr valreg
(reg:OO 66 2)

For the target, it's pseudo and able to get subreg:SI; while for the valreg, it's hard reg, fails to gen subreg, but it will call operand_subword_force (y, i, mode) to get subword further, it goes with:

   copy_to_reg (op);

     further call: emit_move_insn (temp, x) 

// back to the original, OOmode move from the hard register to another pseudo, and again and again... until memory run out for allocation.

If the answer to the question above is it's still meaningful to expand this call without an expected context, I think we have to extend OOmode and XOmode handling.

Any thoughts?

You miss all extra errors the expand_call can generate.  This is the general
reason why we try to continue instead of stopping after the first error.  The
reason is that later errors may be more obvious to the user.  This of course
does no longer work so well because our errors now take 30 lines instead of 1.

It probably is best if the generic opaque-mode emit_move code does not try
to move it via some other mode_class.  Peter?

Failing that, we can work around it by having move patterns for those modes
always, but hard erroring on them (FAIL is no good).

(In reply to Segher Boessenkool from comment #4)
> You miss all extra errors the expand_call can generate.  This is the general
> reason why we try to continue instead of stopping after the first error.  The
> reason is that later errors may be more obvious to the user.  This of course
> does no longer work so well because our errors now take 30 lines instead of
> 1.
> 

Thanks for the explanation! One consequent question is that this point can be applied for the other places where some expected conditions don't hold for bif expansion, but I saw the other places are using "return const0_rtx". Is there something special causing this difference?

> It probably is best if the generic opaque-mode emit_move code does not try
> to move it via some other mode_class.  Peter?
> 
> Failing that, we can work around it by having move patterns for those modes
> always, but hard erroring on them (FAIL is no good).

Yeah, one workround can help the ICE gone: (similar thing needed for XOmode as well):

diff --git a/gcc/config/rs6000/mma.md b/gcc/config/rs6000/mma.md
index 907c9d6d516..04e887ad147 100644
--- a/gcc/config/rs6000/mma.md
+++ b/gcc/config/rs6000/mma.md
@@ -268,10 +268,12 @@ (define_int_attr avvi4i4i4 [(UNSPEC_MMA_PMXVI8GER4PP       "pmxvi8ger4pp")
 (define_expand "movoo"
   [(set (match_operand:OO 0 "nonimmediate_operand")
         (match_operand:OO 1 "input_operand"))]
-  "TARGET_MMA"
+  ""
 {
-  rs6000_emit_move (operands[0], operands[1], OOmode);
-  DONE;
+  if (TARGET_MMA) {
+    rs6000_emit_move (operands[0], operands[1], OOmode);
+    DONE;
+  }
 })

 (define_insn_and_split "*movoo"

*** Bug 101323 has been marked as a duplicate of this bug. ***

(In reply to Kewen Lin from comment #5)
> (In reply to Segher Boessenkool from comment #4)
> > You miss all extra errors the expand_call can generate.  This is the general
> > reason why we try to continue instead of stopping after the first error.  The
> > reason is that later errors may be more obvious to the user.  This of course
> > does no longer work so well because our errors now take 30 lines instead of
> > 1.
> 
> Thanks for the explanation! One consequent question is that this point can
> be applied for the other places where some expected conditions don't hold
> for bif expansion, but I saw the other places are using "return const0_rtx".
> Is there something special causing this difference?

Not really, no.  In general we try to continue a bit longer (like, evaluate
the arguments, as here).  This gives much better diagnostics to the user.  In
a few cases you just have to give up early though, for practical reasons.

> > It probably is best if the generic opaque-mode emit_move code does not try
> > to move it via some other mode_class.  Peter?
> > 
> > Failing that, we can work around it by having move patterns for those modes
> > always, but hard erroring on them (FAIL is no good).
> 
> Yeah, one workround can help the ICE gone: (similar thing needed for XOmode
> as well):
> 
> diff --git a/gcc/config/rs6000/mma.md b/gcc/config/rs6000/mma.md
> index 907c9d6d516..04e887ad147 100644
> --- a/gcc/config/rs6000/mma.md
> +++ b/gcc/config/rs6000/mma.md
> @@ -268,10 +268,12 @@ (define_int_attr avvi4i4i4 [(UNSPEC_MMA_PMXVI8GER4PP  
> "pmxvi8ger4pp")
>  (define_expand "movoo"
>    [(set (match_operand:OO 0 "nonimmediate_operand")
>          (match_operand:OO 1 "input_operand"))]
> -  "TARGET_MMA"
> +  ""
>  {
> -  rs6000_emit_move (operands[0], operands[1], OOmode);
> -  DONE;
> +  if (TARGET_MMA) {
> +    rs6000_emit_move (operands[0], operands[1], OOmode);
> +    DONE;
> +  }
>  })

Like that.  But with a big fat comment, what is done when !TARGET_MMA, and
why we do that.  It is arguably the completely wrong thing to do for opaque
modes.

The master branch has been updated by Kewen Lin <linkw@gcc.gnu.org>:

https://gcc.gnu.org/g:9367e3a65f874dffc8f8a3b6760e77fd9ed67117

commit r13-2062-g9367e3a65f874dffc8f8a3b6760e77fd9ed67117
Author: Kewen.Lin <linkw@gcc.gnu.org>
Date:   Tue Aug 16 00:24:07 2022 -0500

    rs6000: Adjust mov optabs for opaque modes [PR103353]
    
    As PR103353 shows, we may want to continue to expand built-in
    function __builtin_vsx_lxvp, even if we have already emitted
    error messages about some missing required conditions.  As
    shown in that PR, without one explicit mov optab on OOmode
    provided, it would call emit_move_insn recursively.
    
    So this patch is to allow the mov pattern to be generated during
    expanding phase if compiler has already seen errors.
    
            PR target/103353
    
    gcc/ChangeLog:
    
            * config/rs6000/mma.md (define_expand movoo): Move TARGET_MMA condition
            check to preparation statements and add handlings for !TARGET_MMA.
            (define_expand movxo): Likewise.
    
    gcc/testsuite/ChangeLog:
    
            * gcc.target/powerpc/pr103353.c: New test.

The releases/gcc-12 branch has been updated by Kewen Lin <linkw@gcc.gnu.org>:

https://gcc.gnu.org/g:d0d72e0b1ebbac487d70281a56799bf547034ec1

commit r12-8709-gd0d72e0b1ebbac487d70281a56799bf547034ec1
Author: Kewen.Lin <linkw@gcc.gnu.org>
Date:   Tue Aug 16 00:24:07 2022 -0500

    rs6000: Adjust mov optabs for opaque modes [PR103353]
    
    As PR103353 shows, we may want to continue to expand built-in
    function __builtin_vsx_lxvp, even if we have already emitted
    error messages about some missing required conditions.  As
    shown in that PR, without one explicit mov optab on OOmode
    provided, it would call emit_move_insn recursively.
    
    So this patch is to allow the mov pattern to be generated during
    expanding phase if compiler has already seen errors.
    
            PR target/103353
    
    gcc/ChangeLog:
    
            * config/rs6000/mma.md (define_expand movoo): Move TARGET_MMA condition
            check to preparation statements and add handlings for !TARGET_MMA.
            (define_expand movxo): Likewise.
    
    gcc/testsuite/ChangeLog:
    
            * gcc.target/powerpc/pr103353.c: New test.
    
    (cherry picked from commit 9367e3a65f874dffc8f8a3b6760e77fd9ed67117)

The releases/gcc-11 branch has been updated by Kewen Lin <linkw@gcc.gnu.org>:

https://gcc.gnu.org/g:3fc2a9dba4c06115daaa8a39acffadbc33b51152

commit r11-10220-g3fc2a9dba4c06115daaa8a39acffadbc33b51152
Author: Kewen.Lin <linkw@gcc.gnu.org>
Date:   Tue Aug 16 00:24:07 2022 -0500

    rs6000: Adjust mov optabs for opaque modes [PR103353]
    
    As PR103353 shows, we may want to continue to expand built-in
    function __builtin_vsx_lxvp, even if we have already emitted
    error messages about some missing required conditions.  As
    shown in that PR, without one explicit mov optab on OOmode
    provided, it would call emit_move_insn recursively.
    
    So this patch is to allow the mov pattern to be generated during
    expanding phase if compiler has already seen errors.
    
            PR target/103353
    
    gcc/ChangeLog:
    
            * config/rs6000/mma.md (define_expand movoo): Move TARGET_MMA condition
            check to preparation statements and add handlings for !TARGET_MMA.
            (define_expand movxo): Likewise.
    
    gcc/testsuite/ChangeLog:
    
            * gcc.target/powerpc/pr103353.c: New test.
    
    (cherry picked from commit 9367e3a65f874dffc8f8a3b6760e77fd9ed67117)

The releases/gcc-10 branch has been updated by Kewen Lin <linkw@gcc.gnu.org>:

https://gcc.gnu.org/g:501fba663052b0b4a1eb6a42c32b74c254cbedbc

commit r10-10959-g501fba663052b0b4a1eb6a42c32b74c254cbedbc
Author: Kewen Lin <linkw@linux.ibm.com>
Date:   Tue Aug 23 03:31:17 2022 -0500

    rs6000: Adjust mov optabs for opaque modes [PR103353]
    
    As PR103353 shows, we may want to continue to expand built-in
    function __builtin_vsx_lxvp, even if we have already emitted
    error messages about some missing required conditions.  As
    shown in that PR, without one explicit mov optab on OOmode
    provided, it would call emit_move_insn recursively.
    
    So this patch is to allow the mov pattern to be generated during
    expanding phase if compiler has already seen errors.
    
            PR target/103353
    
    gcc/ChangeLog:
    
            * config/rs6000/mma.md (define_expand movpoi): Move TARGET_MMA condition
            check to preparation statements and add handlings for !TARGET_MMA.
            (define_expand movpxi): Likewise.
    
    gcc/testsuite/ChangeLog:
    
            * gcc.target/powerpc/pr103353.c: New test.
    
    (cherry picked from commit 9367e3a65f874dffc8f8a3b6760e77fd9ed67117)

Fixed everywhere.