[PATCH] Don't change DR_STMT if vect_pattern_recog_1 would fail (PR tree-optimization/51356)

[Jakub Jelinek]

Hi!

As mentioned in the PR, vect_pattern_recog_1 attempts to find out
if the computed type_in and type_out are already vector types or not,
and uses VECTOR_MODE_P (TYPE_MODE (type_in)) as the test.  Unfortunately,
get_vectype_for_scalar_type on some targets (e.g. PowerPC) returns a
VECTOR_TYPE with TImode for a DImode integer/boolean scalar type.
If that happens, vect_recog_bool_pattern assumes it will succeed and changes
DR_STMT, but vect_mark_pattern_stmts isn't called and we ICE later on.
Not sure what actually can be vectorized using scalar mode vectors,
so either we adjust vect_recog_bool_pattern like this, or perhaps
vect_pattern_recog_1 could use a different test (TREE_CODE (type_in) ==
VECTOR_TYPE)?

This has been bootstrapped/regtested on x86_64-linux and i686-linux
and fixes the testcase on PowerPC.

2011-12-01  Jakub Jelinek  <jakub@redhat.com>

	PR tree-optimization/51356
	* tree-vect-patterns.c (vect_recog_bool_pattern): Give up if
	vectype doesn't have VECTOR_MODE_P.

--- gcc/tree-vect-patterns.c.jj	2011-11-29 15:09:18.000000000 +0100
+++ gcc/tree-vect-patterns.c	2011-11-30 17:57:42.183149742 +0100
@@ -2078,6 +2078,8 @@ vect_recog_bool_pattern (VEC (gimple, he
       stmt_vec_info pattern_stmt_info;
       vectype = STMT_VINFO_VECTYPE (stmt_vinfo);
       gcc_assert (vectype != NULL_TREE);
+      if (!VECTOR_MODE_P (TYPE_MODE (vectype)))
+	return NULL;
       if (!check_bool_pattern (var, loop_vinfo))
 	return NULL;
 

	Jakub