[Bug tree-optimization/107096] Fully masking vectorization with AVX512 ICEs gcc.dg/vect/vect-over-widen-*.c

[rguenth at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=107096

--- Comment #9 from Richard Biener <rguenth at gcc dot gnu.org> ---
(In reply to rsandifo@gcc.gnu.org from comment #8)
> (In reply to rguenther@suse.de from comment #7)
> > more like precision but x86 uses QImode for two-element, four-element
> > and eight-element masks (rather than two partial integer modes with
> > two and four bits precision).
> Ah, OK.  So yeah, maybe the precision of the vector boolean element *
> the number of elements.

For SVE the following holds:

diff --git a/gcc/tree-vect-loop.cc b/gcc/tree-vect-loop.cc
index 1996ecfee7a..9b24b481867 100644
--- a/gcc/tree-vect-loop.cc
+++ b/gcc/tree-vect-loop.cc
@@ -10097,6 +10097,12 @@ vect_get_loop_mask (gimple_stmt_iterator *gsi,
vec_loop_masks *masks,
                              TYPE_VECTOR_SUBPARTS (vectype)));
       gimple_seq seq = NULL;
       mask_type = truth_type_for (vectype);
+      /* Assert that both mask types have the same total number of value
+        bits.  */
+      gcc_assert (known_eq (TYPE_PRECISION (TREE_TYPE (TREE_TYPE (mask)))
+                           * TYPE_VECTOR_SUBPARTS (TREE_TYPE (mask)),
+                           TYPE_PRECISION (TREE_TYPE (mask_type))
+                           * TYPE_VECTOR_SUBPARTS (mask_type)));
       mask = gimple_build (&seq, VIEW_CONVERT_EXPR, mask_type, mask);
       if (seq)
        gsi_insert_seq_before (gsi, seq, GSI_SAME_STMT);

for AVX the TYPE_PRECISION is always 1 so for unequal subparts we cannot
directly share masks.

I'm going to change LOOP_VINFO_MASKS from an array indexed by nV to
a two-dimensional indexed by nV and bit-precision * subparts.  Well,
probably using a hash_map instead since this will be quite sparse.
Or maybe not, but at least dynamically growing as we do now is difficult
and subparts can be non-constant.