[Bug target/98478] New: AVX512 refactoring integer mask with VnBImode

[crazylht at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=98478

            Bug ID: 98478
           Summary: AVX512 refactoring integer mask with VnBImode
           Product: gcc
           Version: 11.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: crazylht at gmail dot com
                CC: hjl.tools at gmail dot com, wwwhhhyyy333 at gmail dot com
  Target Milestone: ---
            Target: x86_64-*-* i?86-*-*

When simplily add the bellow
-----
+/* For AVX512 Integer mask.  */
+VECTOR_BOOL_MODE (V64BI, 64, 8);
+VECTOR_BOOL_MODE (V32BI, 32, 4);
+VECTOR_BOOL_MODE (V16BI, 16, 2);
+VECTOR_BOOL_MODE (V8BI, 8, 1);
+VECTOR_BOOL_MODE (V4BI, 4, 1);
+VECTOR_BOOL_MODE (V2BI, 2, 1);
-----

GCC failed self-test, see below

-----
 simplify_gen_subreg (inner_mode, vm, mode, offset))
  expected: (reg:BI 102)
  actual: (nil)

cc1: internal compiler error: in assert_rtx_eq_at, at selftest-rtl.c:57
0x142f202 selftest::assert_rtx_eq_at(selftest::location const&, char const*,
rtx_def*, rtx_def*)
       
/export/users2/liuhongt/gcc/gnu-toolchain/master_debug/gcc/selftest-rtl.c:57
0x1459aef test_vector_ops_duplicate
       
/export/users2/liuhongt/gcc/gnu-toolchain/master_debug/gcc/simplify-rtx.c:7552
0x145cddd test_vector_ops
       
/export/users2/liuhongt/gcc/gnu-toolchain/master_debug/gcc/simplify-rtx.c:7947
0x145ce98 selftest::simplify_rtx_c_tests()
       
/export/users2/liuhongt/gcc/gnu-toolchain/master_debug/gcc/simplify-rtx.c:8012
0x272818f selftest::run_tests()
       
/export/users2/liuhongt/gcc/gnu-toolchain/master_debug/gcc/selftest-run-tests.c:118
----


SVE didn't fail this because the nunits is not constant and won't go in this
test.

  if (nunits.is_constant (&const_nunits))
    {
      /* And again with the final element.  */
      rtx last_index = gen_int_mode (const_nunits - 1, word_mode);
      rtx last_par = gen_rtx_PARALLEL (VOIDmode, gen_rtvec (1, last_index));
      ASSERT_RTX_PTR_EQ (scalar_reg,
                         simplify_binary_operation (VEC_SELECT, inner_mode,
                                                    duplicate, last_par));

      /* Test a scalar subreg of a VEC_MERGE of a VEC_DUPLICATE.  */
      rtx vector_reg = make_test_reg (mode);
      for (unsigned HOST_WIDE_INT i = 0; i < const_nunits; i++)
        {
          if (i >= HOST_BITS_PER_WIDE_INT)
            break;
          rtx mask = GEN_INT ((HOST_WIDE_INT_1U << i) | (i + 1));
          rtx vm = gen_rtx_VEC_MERGE (mode, duplicate, vector_reg, mask);
          poly_uint64 offset = i * GET_MODE_SIZE (inner_mode);
          ASSERT_RTX_EQ (scalar_reg,
                         simplify_gen_subreg (inner_mode, vm,
                                              mode, offset));
        }
    }


it failed to get simplify_gen_subreg (BImode, vm, V16BImode, offset), also
offset doesn't seems right.