[Bug tree-optimization/95487] [10/11 Regression] ICE: verify_gimple failed (error: invalid vector types in nop conversion) with -O3 -march=skylake-avx512 since r10-1052-gc29c92c789d93848

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=95487

--- Comment #5 from Richard Biener <rguenth at gcc dot gnu.org> ---
We also fail to unswitch the outer loop on if (d) (probably simply because we
don't unswitch outer loops).  It's likely the invariantness of the condition
that makes the problem.  We're vectorizing

  <bb 32> [local count: 437450365]:
  # b_43 = PHI <b_24(33), b_15(53)>
  _46 = (int) b_43;
  _65 = &g[_46];
  .MASK_STORE (_65, 32B, d.0_2, 0);
  b.3_25 = (unsigned short) b_43;
  _33 = b.3_25 + 2;
  b_24 = (short int) _33;
  if (b_24 >= 0)
    goto <bb 30>; [11.00%]
  else
    goto <bb 33>; [89.00%]

  <bb 33> [local count: 389330825]:
  goto <bb 32>; [100.00%]

where we use a scatter because we fail to analyze the evolution for g[b].
We don't get a vector boolean type for the mask because the "magic" here
doesn't work this way.

1508          else if (VECT_SCALAR_BOOLEAN_TYPE_P (TREE_TYPE (op))
1509                   && VECTOR_BOOLEAN_TYPE_P (stmt_vectype))
1510            vector_type = truth_type_for (stmt_vectype);

which does not consider that the mask in scatters might need a
vector boolean type.  But it appearantly does.

Test coverage for this code seems to be non-existent though :/

I do have a patch that fixes the testcase though.