[Bug target/78938] [7 Regression] ICE in expand_vec_cond_expr, at optabs.c:5636 w/ -mavx512bw -ftree-loop-vectorize -O1

[rguenther at suse dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78938

--- Comment #3 from rguenther at suse dot de <rguenther at suse dot de> ---
On Mon, 2 Jan 2017, jakub at gcc dot gnu.org wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=78938
> 
> Jakub Jelinek <jakub at gcc dot gnu.org> changed:
> 
>            What    |Removed                     |Added
> ----------------------------------------------------------------------------
>                  CC|                            |rguenth at gcc dot gnu.org
> 
> --- Comment #2 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
> vectorizable_comparison has complicated code to deal with comparisons of
> VECTOR_BOOLEAN_P operands, but vectorizable_condition does not have any of
> this.
> So, IMHO either we duplicate that (or partially move into helper functions),
> e.g. the
>   /* Boolean values may have another representation in vectors
>      and therefore we prefer bit operations over comparison for
>      them (which also works for scalar masks).  We store opcodes
>      to use in bitop1 and bitop2.  Statement is vectorized as
>        BITOP2 (rhs1 BITOP1 rhs2) or
>        rhs1 BITOP2 (BITOP1 rhs2)
>      depending on bitop1 and bitop2 arity.  */
>   if (VECTOR_BOOLEAN_TYPE_P (vectype))
> stuff and the bitop1/bitop2 handling later on, or refuse to vectorize COND_EXPR
> if the comparison operands are VECTOR_BOOLEAN_TYPE_P (with the exception of
> easy cases where rhs1 is the boolean itself or something easily transformable
> into that (boolv != 0, boolv != 1, boolv == 0, boolv == 1) perhaps with
> swapping rhs2 and rhs3.  Or refuse to vectorize such COND_EXPRs and add pattern
> recognizer that replaces x = boolv1 op boolv2 ? rhs2 : rhs3; with
> tmp = boolv1 op boolv2;
> x = tmp ? rhs2 : rhs3;
> and then vectorizable_comparison should be able to deal with the first stmt and
> vectorizable_condition with the latter.
> Richard, any preferences?

I believe the bool patterns were supposed to handle this case but
at the time I fixed its time/size complexity issues we IIRC ended up
effectively disabling them for x86 with no fallout -- this may be
the fallout.  So I'd figure out why the bool patterns stuff does not
trigger here first.  IIRC the "effective disabling" was due to
some target optab/hook check that always returns true on x86
(and thus possibly needs to be changed to check for a bad "mix" of
VECTOR_BOOLEAN kinds)