[patch] vectorizer cleanups (and prep for outer-loop vectorization)
Dorit Nuzman
DORIT@il.ibm.com
Wed Apr 11 11:51:00 GMT 2007
Zdenek Dvorak <rakdver@atrey.karlin.mff.cuni.cz> wrote on 02/04/2007
12:48:12:
> Hello,
>
> I will look at the patch. However, would it be possible to split it
> into smaller parts? The changes seem to be quite independent,
>
coming up...
thanks,
dorit
> Zdenek
>
> > This is exactly the same patch as was committed to autovect last week:
> > http://gcc.gnu.org/ml/gcc-patches/2007-03/msg01904.html
> > (but relative to mainline, and after testig on mainline).
> >
> > These are a few cleanups (really technical things, no new
functionality),
> > some of which will be also useful for outer-loop vectorization:
> >
> > o Remove the field exit_cond from loop_vec_info, which used to hold the
> > loop-exit stmt. Instead, a stmt that is identified as a loop-exit
control
> > stmt is marked as a 'loop_exit_ctrl_vec_info_type' (new enum value for
> > stmt_vec_info_type). This will be useful when looking at outer-loops
> > because we have more than one stmt like that in the loop (there's also
one
> > in the inner-loop).
> >
> > o Vectorization of induction is currently done upon encountering a use
of
> > the induction variable. Instead, we now vectorize the induction
earlier,
> > when we encounter the respective induction phi-node. This simplifies
things
> > a bit cause it's more in-line with how we vectorize other things (when
we
> > encounter a use of the induction we just get the vector def from the
> > STMT_VINFO_VEC_STMT field of the defining phi-node), and also we don't
need
> > to worry about vectorizing the same induction multiple times. When
looking
> > at outer-loops this will be even more useful cause when we vectorize a
> > phi-node we know exactly relative to which loop-nest is the induction,
and
> > don't need any extra logic to check for that.
> >
> > o Reorganize the calls to the vectorizable_* functions: In the drivers
that
> > call these functions (vect_analyze_operations and vect_transform_stmt),
we
> > currently call most of these functions only for RELEVANT_P stmts, and
then
> > call vectorizable_reduction only for LIVE_P stmts. Now, instead, each
of
> > the vectorizable_* functions checks that STMT_VINFO_RELEVANT/LIVE_P are
as
> > expected, and in the two driver functions we don't check for
> > STMT_VINFO_RELEVANT_P anymore, but apply the vectorizable_* functions
to
> > all stmts. When looking at outer-loops this is useful because a
reduction
> > (in the inner-loop) may now be marked RELEVANT (if it is used in the
> > outer-loop), and not only LIVE as it is currently.
> >
> > o A bunch of other cleanups (that are not necessarily inspired by
> > outer-loop vectorization): Factor out code from
> > vect_mark_stmts_to_be_vectorized to process_use; Remove the '#ifdef
> > ADJUST_IN_EPILOG' and the code path that was disabled by it (which
removes
> > unused code and simplifies the function); Add a flag to allow disabling
> > tree-reassociation, to help test reductions that are otherwise
transformed
> > by reassociation to something we currently don't recognize; Fix
> > indentation; A few ther small miscellaneous stuff.
> >
> > Bootstrapped and tested (full regression testing) on powerpc64-linux
and
> > i386-linux.
> > Also bootstrapped with vectorization enabled on powerpc64-linux.
> >
> > ok for mainline?
> >
> > :ADDPATCH vectorizer,non-algorithmic:
> >
> > ChangeLog:
> >
> > * tree-vectorizer.h (_loop_vec_info): Remove unused field
> > exit_cond.
> > (stmt_vec_info_type): Add enum value
loop_exit_ctrl_vec_info_type.
> > * tree-vectorizer.c (new_loop_vec_info): Remove setting of
> > LOOP_VINFO_EXIT_COND.
> > * tree-vect-analyze.c (vect_stmt_relevant_p): Replace use of
> > LOOP_VINFO_EXIT_COND with use of loop_exit_ctrl_vec_info_type.
> > (vect_analyze_loop_form): Likewise.
> >
> > * tree-vectorizer.h (stmt_vec_info_type): Add enum value
> > induc_vec_info_type.
> > (vectorizable_induction): New function declaration.
> > * tree-vect-transform.c (get_initial_def_for_induction): No
need to
> > check if already vectorized. Find first place in BB where nre
> > stmts
> > can be inserted.
> > (vectorizable_induction): New function.
> > (vect_transform_stmt): Add case for induc_vec_info_type to call
> > vectorizable_induction.
> > (vect_transform_loop): Consider phis for vectorization.
> > * tree-vect-analyze.c (analyze_operations): Call
> > vectorizable_induction when analyzing phis and when analyzing
> > stmts.
> >
> > * tree-vectorizer.c (vect_is_simple_use): Remove redundant
check.
> > (destroy_loop_vec_info): Set loop->aux to NULL.
> >
> > * tree-vect-analyze.c (vect_determine_vectorization_factor):
> > Simplify
> > condition. Add dump print.
> > (vect_analyze_operations): Fix indenetation. Fix a comment.
Fix a
> > print message.
> > (vect_analyze_scalar_cycles): Fix indentation.
> > (vect_enhance_data_refs_alignment): Fix check in case of
peeling.
> > (vect_mark_relevant): Include phis in relevance analysis.
> > (process_use): New function.
> > (vect_mark_stmts_to_be_vectorized): Factor out code to
process_use.
> > Check phis in all bbs.
> >
> > * tree-vect-patterns.c (widened_name_p): Remove redundant
check.
> >
> > * tree-vect-analyze.c (vect_analyze_operations): Reorganize
calls
> > to
> > vectorizable_* functions.
> > * tree-vect-transform.c (vectorizable_call): Add check for
> > STMT_VINFO_RELEVANT_P, STMT_VINFO_DEF_TYPE and
STMT_VINFO_LIVE_P.
> > (vectorizable_store): likewise.
> > (vectorizable_conversion): Add check for STMT_VINFO_DEF_TYPE.
> > Add comments.
> > (vectorizable_operation, vectorizable_type_demotion): Likewise.
> > (vectorizable_type_promotion, vectorizable_load): Likewise.
> > (vectorizable_live_operation, vectorizable_condition):
Likewise.
> > (vectorizable_assignment): Add check for STMT_VINFO_DEF_TYPE
and
> > STMT_VINFO_LIVE_P.
> > (vect_transform_stmt): Reorganize calls to vectorizable_*
> > functions.
> > (vect_analyze_loop_form): Use the new NITERS_KNOWN_P macro so
that
> > this check could be done without creating a loop_vinfo first.
> > Record
> > loop_vinfo in loop->aux.
> >
> > * tree-vect-transform.c (get_initial_def_for_reduction): Clean
away
> > the unused code for reduction without adjust-in-epilog to
simplify
> > the
> > function.
> >
> > * tree-vect-transform.c (vect_transform_loop): Add an assert.
> >
> > * tree-ssa-reassoc.c (gate_reassoc): New function.
> > * common.opt (tree-reassoc): New flag.
> > (pass_reassoc): Add gate.
> >
> >
> > (See attached file: mainline-vect-cleanups.mar27.txt)
> > Index: ChangeLog
> > ===================================================================
> > *** ChangeLog (revision 123103)
> > --- ChangeLog (working copy)
> > ***************
> > *** 1,3 ****
> > --- 1,70 ----
> > + 2007-03-20 Dorit Nuzman <dorit@il.ibm.com>
> > +
> > + * tree-vectorizer.h (_loop_vec_info): Remove unused field
exit_cond.
> > + (stmt_vec_info_type): Add enum value loop_exit_ctrl_vec_info_type.
> > + * tree-vectorizer.c (new_loop_vec_info): Remove setting of
> > + LOOP_VINFO_EXIT_COND.
> > + * tree-vect-analyze.c (vect_stmt_relevant_p): Replace use of
> > + LOOP_VINFO_EXIT_COND with use of loop_exit_ctrl_vec_info_type.
> > + (vect_analyze_loop_form): Likewise.
> > +
> > + * tree-vectorizer.h (stmt_vec_info_type): Add enum value
> > + induc_vec_info_type.
> > + (vectorizable_induction): New function declaration.
> > + * tree-vect-transform.c (get_initial_def_for_induction): No need
to
> > + check if already vectorized. Find first place in BB where nre
stmts
> > + can be inserted.
> > + (vectorizable_induction): New function.
> > + (vect_transform_stmt): Add case for induc_vec_info_type to call
> > + vectorizable_induction.
> > + (vect_transform_loop): Consider phis for vectorization.
> > + * tree-vect-analyze.c (analyze_operations): Call
> > + vectorizable_induction when analyzing phis and when analyzing
> > + stmts.
> > +
> > + * tree-vectorizer.c (vect_is_simple_use): Remove redundant
check.
> > + (destroy_loop_vec_info): Set loop->aux to NULL.
> > +
> > + * tree-vect-analyze.c (vect_determine_vectorization_factor):
Simplify
> > + condition. Add dump print.
> > + (vect_analyze_operations): Fix indenetation. Fix a comment. Fix
a
> > + print message.
> > + (vect_analyze_scalar_cycles): Fix indentation.
> > + (vect_enhance_data_refs_alignment): Fix check in case of peeling.
> > + (vect_mark_relevant): Include phis in relevance analysis.
> > + (process_use): New function.
> > + (vect_mark_stmts_to_be_vectorized): Factor out code to
process_use.
> > + Check phis in all bbs.
> > +
> > + * tree-vect-patterns.c (widened_name_p): Remove redundant
check.
> > +
> > + * tree-vect-analyze.c (vect_analyze_operations): Reorganize calls
to
> > + vectorizable_* functions.
> > + * tree-vect-transform.c (vectorizable_call): Add check for
> > + STMT_VINFO_RELEVANT_P, STMT_VINFO_DEF_TYPE and STMT_VINFO_LIVE_P.
> > + (vectorizable_store): likewise.
> > + (vectorizable_conversion): Add check for STMT_VINFO_DEF_TYPE.
> > + Add comments.
> > + (vectorizable_operation, vectorizable_type_demotion): Likewise.
> > + (vectorizable_type_promotion, vectorizable_load): Likewise.
> > + (vectorizable_live_operation, vectorizable_condition): Likewise.
> > + (vectorizable_assignment): Add check for STMT_VINFO_DEF_TYPE and
> > + STMT_VINFO_LIVE_P.
> > + (vect_transform_stmt): Reorganize calls to vectorizable_*
functions.
> > + (vect_analyze_loop_form): Use the new NITERS_KNOWN_P macro so that
> > + this check could be done without creating a loop_vinfo first.
Record
> > + loop_vinfo in loop->aux.
> > +
> > + * tree-vect-transform.c (get_initial_def_for_reduction): Clean
away
> > + the unused code for reduction without adjust-in-epilog to simplify
the
> > + function.
> > +
> > + * tree-vect-transform.c (vect_transform_loop): Add an assert.
> > +
> > + * tree-ssa-reassoc.c (gate_reassoc): New function.
> > + * common.opt (tree-reassoc): New flag.
> > + (pass_reassoc): Add gate.
> > +
> > 2007-03-20 Mark Mitchell <mark@codesourcery.com>
> >
> > * config/arm/elf.h (TARGET_ASM_DESTRUCTOR): Define.
> > Index: testsuite/gcc.dg/vect/vect-1.c
> > ===================================================================
> > *** testsuite/gcc.dg/vect/vect-1.c (revision 123103)
> > --- testsuite/gcc.dg/vect/vect-1.c (working copy)
> > *************** foo (int n)
> > *** 38,44 ****
> > inner-loop: cross iteration cycle; multi-dimensional arrays).
*/
> > diff = 0;
> > for (i = 0; i < N; i++) {
> > ! for (i = 0; i < N; i++) {
> > diff += (image[i][j] - block[i][j]);
> > }
> > }
> > --- 38,44 ----
> > inner-loop: cross iteration cycle; multi-dimensional arrays).
*/
> > diff = 0;
> > for (i = 0; i < N; i++) {
> > ! for (j = 0; j < N; j++) {
> > diff += (image[i][j] - block[i][j]);
> > }
> > }
> > Index: testsuite/gcc.dg/vect/vect.exp
> > ===================================================================
> > *** testsuite/gcc.dg/vect/vect.exp (revision 123103)
> > --- testsuite/gcc.dg/vect/vect.exp (working copy)
> > *************** dg-runtest [lsort [glob -nocomplain $src
> > *** 169,174 ****
> > --- 169,175 ----
> > "" $DEFAULT_VECTCFLAGS
> >
> > # With -Os
> > + set DEFAULT_VECTCFLAGS $SAVED_DEFAULT_VECTCFLAGS
> > lappend DEFAULT_VECTCFLAGS "-Os"
> > dg-runtest [lsort [glob -nocomplain
$srcdir/$subdir/Os-vect-*.\[cS\]]] \
> > "" $DEFAULT_VECTCFLAGS
> > Index: tree-vectorizer.c
> > ===================================================================
> > *** tree-vectorizer.c (revision 123103)
> > --- tree-vectorizer.c (working copy)
> > *************** new_loop_vec_info (struct loop *loop)
> > *** 1420,1426 ****
> >
> > LOOP_VINFO_LOOP (res) = loop;
> > LOOP_VINFO_BBS (res) = bbs;
> > - LOOP_VINFO_EXIT_COND (res) = NULL;
> > LOOP_VINFO_NITERS (res) = NULL;
> > LOOP_VINFO_VECTORIZABLE_P (res) = 0;
> > LOOP_PEELING_FOR_ALIGNMENT (res) = 0;
> > --- 1420,1425 ----
> > *************** destroy_loop_vec_info (loop_vec_info loo
> > *** 1511,1516 ****
> > --- 1510,1516 ----
> > VEC_free (tree, heap, LOOP_VINFO_MAY_MISALIGN_STMTS (loop_vinfo));
> >
> > free (loop_vinfo);
> > + loop->aux = NULL;
> > }
> >
> >
> > *************** vect_is_simple_use (tree operand, loop_v
> > *** 1714,1728 ****
> > return false;
> > }
> >
> > - /* stmts inside the loop that have been identified as performing
> > - a reduction operation cannot have uses in the loop. */
> > - if (*dt == vect_reduction_def && TREE_CODE (*def_stmt) != PHI_NODE)
> > - {
> > - if (vect_print_dump_info (REPORT_DETAILS))
> > - fprintf (vect_dump, "reduction used in loop.");
> > - return false;
> > - }
> > -
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "type of def: %d.",*dt);
> >
> > --- 1714,1719 ----
> > *************** vect_is_simple_use (tree operand, loop_v
> > *** 1730,1742 ****
> > {
> > case PHI_NODE:
> > *def = PHI_RESULT (*def_stmt);
> > - gcc_assert (*dt == vect_induction_def || *dt ==
vect_reduction_def
> > - || *dt == vect_invariant_def);
> > break;
> >
> > case GIMPLE_MODIFY_STMT:
> > *def = GIMPLE_STMT_OPERAND (*def_stmt, 0);
> > - gcc_assert (*dt == vect_loop_def || *dt == vect_invariant_def);
> > break;
> >
> > default:
> > --- 1721,1730 ----
> > Index: tree-vectorizer.h
> > ===================================================================
> > *** tree-vectorizer.h (revision 123103)
> > --- tree-vectorizer.h (working copy)
> > *************** typedef struct _loop_vec_info {
> > *** 93,101 ****
> > /* The loop basic blocks. */
> > basic_block *bbs;
> >
> > - /* The loop exit_condition. */
> > - tree exit_cond;
> > -
> > /* Number of iterations. */
> > tree num_iters;
> >
> > --- 93,98 ----
> > *************** typedef struct _loop_vec_info {
> > *** 138,144 ****
> > /* Access Functions. */
> > #define LOOP_VINFO_LOOP(L) (L)->loop
> > #define LOOP_VINFO_BBS(L) (L)->bbs
> > - #define LOOP_VINFO_EXIT_COND(L) (L)->exit_cond
> > #define LOOP_VINFO_NITERS(L) (L)->num_iters
> > #define LOOP_VINFO_VECTORIZABLE_P(L) (L)->vectorizable
> > #define LOOP_VINFO_VECT_FACTOR(L) (L)->vectorization_factor
> > --- 135,140 ----
> > *************** typedef struct _loop_vec_info {
> > *** 151,159 ****
> > #define LOOP_VINFO_MAY_MISALIGN_STMTS(L) (L)->may_misalign_stmts
> > #define LOOP_VINFO_LOC(L) (L)->loop_line_number
> >
> > #define LOOP_VINFO_NITERS_KNOWN_P(L) \
> > ! (host_integerp ((L)->num_iters,0) \
> > ! && TREE_INT_CST_LOW ((L)->num_iters) > 0)
> >
> > /*-----------------------------------------------------------------*/
> > /* Info on vectorized defs. */
> > --- 147,158 ----
> > #define LOOP_VINFO_MAY_MISALIGN_STMTS(L) (L)->may_misalign_stmts
> > #define LOOP_VINFO_LOC(L) (L)->loop_line_number
> >
> > + #define NITERS_KNOWN_P(n) \
> > + (host_integerp ((n),0) \
> > + && TREE_INT_CST_LOW ((n)) > 0)
> > +
> > #define LOOP_VINFO_NITERS_KNOWN_P(L) \
> > ! NITERS_KNOWN_P((L)->num_iters)
> >
> > /*-----------------------------------------------------------------*/
> > /* Info on vectorized defs. */
> > *************** enum stmt_vec_info_type {
> > *** 167,175 ****
> > assignment_vec_info_type,
> > condition_vec_info_type,
> > reduc_vec_info_type,
> > type_promotion_vec_info_type,
> > type_demotion_vec_info_type,
> > ! type_conversion_vec_info_type
> > };
> >
> > /* Indicates whether/how a variable is used in the loop. */
> > --- 166,176 ----
> > assignment_vec_info_type,
> > condition_vec_info_type,
> > reduc_vec_info_type,
> > + induc_vec_info_type,
> > type_promotion_vec_info_type,
> > type_demotion_vec_info_type,
> > ! type_conversion_vec_info_type,
> > ! loop_exit_ctrl_vec_info_type
> > };
> >
> > /* Indicates whether/how a variable is used in the loop. */
> > *************** extern bool vectorizable_call (tree, blo
> > *** 428,433 ****
> > --- 429,435 ----
> > extern bool vectorizable_condition (tree, block_stmt_iterator *, tree
*);
> > extern bool vectorizable_live_operation (tree,
> block_stmt_iterator *, tree *);
> > extern bool vectorizable_reduction (tree, block_stmt_iterator *, tree
*);
> > + extern bool vectorizable_induction (tree, block_stmt_iterator *, tree
*);
> > /* Driver for transformation stage. */
> > extern void vect_transform_loop (loop_vec_info);
> >
> > Index: tree-vect-analyze.c
> > ===================================================================
> > *** tree-vect-analyze.c (revision 123103)
> > --- tree-vect-analyze.c (working copy)
> > *************** vect_determine_vectorization_factor (loo
> > *** 122,134 ****
> >
> > gcc_assert (stmt_info);
> >
> > ! /* Two cases of "relevant" phis: those that define an
> > ! induction that is used in the loop, and those that
> > ! define a reduction. */
> > ! if ((STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
> > ! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
> > ! || (STMT_VINFO_RELEVANT (stmt_info) ==
vect_used_by_reduction
> > ! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_reduction_def))
> > {
> > gcc_assert (!STMT_VINFO_VECTYPE (stmt_info));
> > scalar_type = TREE_TYPE (PHI_RESULT (phi));
> > --- 122,128 ----
> >
> > gcc_assert (stmt_info);
> >
> > ! if (STMT_VINFO_RELEVANT_P (stmt_info))
> > {
> > gcc_assert (!STMT_VINFO_VECTYPE (stmt_info));
> > scalar_type = TREE_TYPE (PHI_RESULT (phi));
> > *************** vect_determine_vectorization_factor (loo
> > *** 268,274 ****
> > }
> >
> > /* TODO: Analyze cost. Decide if worth while to vectorize. */
> > !
> > if (vectorization_factor <= 1)
> > {
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > --- 262,269 ----
> > }
> >
> > /* TODO: Analyze cost. Decide if worth while to vectorize. */
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "vectorization factor = %d",
> vectorization_factor);
> > if (vectorization_factor <= 1)
> > {
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > *************** vect_analyze_operations (loop_vec_info l
> > *** 311,316 ****
> > --- 306,313 ----
> >
> > for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
> > {
> > + ok = true;
> > +
> > stmt_info = vinfo_for_stmt (phi);
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > *************** vect_analyze_operations (loop_vec_info l
> > *** 319,341 ****
> > }
> >
> > gcc_assert (stmt_info);
> > -
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > fprintf (vect_dump, "not vectorized: value used after loop.");
> > ! return false;
> > }
> >
> > if (STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
> > && STMT_VINFO_DEF_TYPE (stmt_info) != vect_induction_def)
> > {
> > ! /* Most likely a reduction-like computation that is used
> > ! in the loop. */
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > ! fprintf (vect_dump, "not vectorized: unsupported
pattern.");
> > ! return false;
> > }
> > }
> >
> > --- 316,351 ----
> > }
> >
> > gcc_assert (stmt_info);
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > fprintf (vect_dump, "not vectorized: value used after loop.");
> > ! return false;
> > }
> >
> > if (STMT_VINFO_RELEVANT (stmt_info) == vect_used_in_loop
> > && STMT_VINFO_DEF_TYPE (stmt_info) != vect_induction_def)
> > {
> > ! /* A scalar-dependence cycle that we don't support. */
> > ! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > ! fprintf (vect_dump, "not vectorized: scalar
> dependence cycle.");
> > ! return false;
> > ! }
> > !
> > ! if (STMT_VINFO_RELEVANT_P (stmt_info)
> > ! && STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
> > ! ok = vectorizable_induction (phi, NULL, NULL);
> > !
> > ! if (!ok)
> > ! {
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > ! {
> > ! fprintf (vect_dump,
> > ! "not vectorized: relevant phi not supported: ");
> > ! print_generic_expr (vect_dump, phi, TDF_SLIM);
> > ! }
> > ! return false;
> > }
> > }
> >
> > *************** vect_analyze_operations (loop_vec_info l
> > *** 343,348 ****
> > --- 353,359 ----
> > {
> > tree stmt = bsi_stmt (si);
> > stmt_vec_info stmt_info = vinfo_for_stmt (stmt);
> > + enum vect_def_type relevance = STMT_VINFO_RELEVANT (stmt_info);
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > *************** vect_analyze_operations (loop_vec_info l
> > *** 367,421 ****
> > continue;
> > }
> >
> > if (STMT_VINFO_RELEVANT_P (stmt_info))
> > {
> > gcc_assert (GIMPLE_STMT_P (stmt)
> > || !VECTOR_MODE_P (TYPE_MODE (TREE_TYPE (stmt))));
> > gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
> > -
> > - ok = (vectorizable_type_promotion (stmt, NULL, NULL)
> > - || vectorizable_type_demotion (stmt, NULL, NULL)
> > - || vectorizable_conversion (stmt, NULL, NULL)
> > - || vectorizable_operation (stmt, NULL, NULL)
> > - || vectorizable_assignment (stmt, NULL, NULL)
> > - || vectorizable_load (stmt, NULL, NULL)
> > - || vectorizable_call (stmt, NULL, NULL)
> > - || vectorizable_store (stmt, NULL, NULL)
> > - || vectorizable_condition (stmt, NULL, NULL));
> > -
> > - if (!ok)
> > - {
> > - if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > - {
> > - fprintf (vect_dump,
> > - "not vectorized: relevant stmt not supported: ");
> > - print_generic_expr (vect_dump, stmt, TDF_SLIM);
> > - }
> > - return false;
> > - }
> > need_to_vectorize = true;
> > ! }
> > !
> > ! if (STMT_VINFO_LIVE_P (stmt_info))
> > ! {
> > ! ok = vectorizable_reduction (stmt, NULL, NULL);
> >
> > ! if (ok)
> > ! need_to_vectorize = true;
> > ! else
> > ! ok = vectorizable_live_operation (stmt, NULL, NULL);
> >
> > ! if (!ok)
> > {
> > ! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > ! {
> > ! fprintf (vect_dump,
> > ! "not vectorized: live stmt not supported: ");
> > ! print_generic_expr (vect_dump, stmt, TDF_SLIM);
> > ! }
> > ! return false;
> > }
> > ! }
> > } /* stmts in bb */
> > } /* bbs */
> >
> > --- 378,434 ----
> > continue;
> > }
> >
> > + switch (STMT_VINFO_DEF_TYPE (stmt_info))
> > + {
> > + case vect_loop_def:
> > + break;
> > +
> > + case vect_reduction_def:
> > + gcc_assert (relevance == vect_unused_in_loop);
> > + break;
> > +
> > + case vect_induction_def:
> > + case vect_constant_def:
> > + case vect_invariant_def:
> > + case vect_unknown_def_type:
> > + default:
> > + gcc_unreachable ();
> > + }
> > +
> > if (STMT_VINFO_RELEVANT_P (stmt_info))
> > {
> > gcc_assert (GIMPLE_STMT_P (stmt)
> > || !VECTOR_MODE_P (TYPE_MODE (TREE_TYPE (stmt))));
> > gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
> > need_to_vectorize = true;
> > ! }
> >
> > ! ok = (vectorizable_type_promotion (stmt, NULL, NULL)
> > ! || vectorizable_type_demotion (stmt, NULL, NULL)
> > ! || vectorizable_conversion (stmt, NULL, NULL)
> > ! || vectorizable_operation (stmt, NULL, NULL)
> > ! || vectorizable_assignment (stmt, NULL, NULL)
> > ! || vectorizable_load (stmt, NULL, NULL)
> > ! || vectorizable_call (stmt, NULL, NULL)
> > ! || vectorizable_store (stmt, NULL, NULL)
> > ! || vectorizable_condition (stmt, NULL, NULL)
> > ! || vectorizable_reduction (stmt, NULL, NULL));
> > !
> > ! /* Stmts that are (also) "live" (i.e. - that are used out
> of the loop)
> > ! need extra handling, except for vectorizable reductions. */
> > ! if (STMT_VINFO_LIVE_P (stmt_info)
> > ! && STMT_VINFO_TYPE (stmt_info) != reduc_vec_info_type)
> > ! ok |= vectorizable_live_operation (stmt, NULL, NULL);
> >
> > ! if (!ok)
> > ! {
> > ! if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > {
> > ! fprintf (vect_dump, "not vectorized: stmt not supported: ");
> > ! print_generic_expr (vect_dump, stmt, TDF_SLIM);
> > }
> > ! return false;
> > ! }
> > } /* stmts in bb */
> > } /* bbs */
> >
> > *************** vect_analyze_scalar_cycles (loop_vec_inf
> > *** 605,618 ****
> > }
> >
> > if (!access_fn
> > ! || !vect_is_simple_iv_evolution (loop->num, access_fn,
> &dumy, &dumy))
> > {
> > ! VEC_safe_push (tree, heap, worklist, phi);
> > continue;
> > }
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "Detected induction.");
> > STMT_VINFO_DEF_TYPE (stmt_vinfo) = vect_induction_def;
> > }
> >
> > --- 618,631 ----
> > }
> >
> > if (!access_fn
> > ! || !vect_is_simple_iv_evolution (loop->num, access_fn, &dumy,
&dumy))
> > {
> > ! VEC_safe_push (tree, heap, worklist, phi);
> > continue;
> > }
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "Detected induction.");
> > STMT_VINFO_DEF_TYPE (stmt_vinfo) = vect_induction_def;
> > }
> >
> > *************** static bool
> > *** 1418,1423 ****
> > --- 1431,1437 ----
> > vect_enhance_data_refs_alignment (loop_vec_info loop_vinfo)
> > {
> > VEC (data_reference_p, heap) *datarefs = LOOP_VINFO_DATAREFS
> (loop_vinfo);
> > + struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> > enum dr_alignment_support supportable_dr_alignment;
> > struct data_reference *dr0 = NULL;
> > struct data_reference *dr;
> > *************** vect_enhance_data_refs_alignment (loop_v
> > *** 1515,1521 ****
> >
> > /* Often peeling for alignment will require peeling for loop-
> bound, which in
> > turn requires that we know how to adjust the loop ivs after
> the loop. */
> > ! if (!vect_can_advance_ivs_p (loop_vinfo))
> > do_peeling = false;
> >
> > if (do_peeling)
> > --- 1529,1536 ----
> >
> > /* Often peeling for alignment will require peeling for loop-
> bound, which in
> > turn requires that we know how to adjust the loop ivs after
> the loop. */
> > ! if (!vect_can_advance_ivs_p (loop_vinfo)
> > ! || !slpeel_can_duplicate_loop_p (loop, single_exit (loop)))
> > do_peeling = false;
> >
> > if (do_peeling)
> > *************** vect_mark_relevant (VEC(tree,heap) **wor
> > *** 2100,2110 ****
> > if (relevant > STMT_VINFO_RELEVANT (stmt_info))
> > STMT_VINFO_RELEVANT (stmt_info) = relevant;
> >
> > - if (TREE_CODE (stmt) == PHI_NODE)
> > - /* Don't put phi-nodes in the worklist. Phis that are marked
relevant
> > - or live will fail vectorization later on. */
> > - return;
> > -
> > if (STMT_VINFO_RELEVANT (stmt_info) == save_relevant
> > && STMT_VINFO_LIVE_P (stmt_info) == save_live_p)
> > {
> > --- 2115,2120 ----
> > *************** vect_stmt_relevant_p (tree stmt, loop_ve
> > *** 2143,2149 ****
> > *live_p = false;
> >
> > /* cond stmt other than loop exit cond. */
> > ! if (is_ctrl_stmt (stmt) && (stmt != LOOP_VINFO_EXIT_COND
(loop_vinfo)))
> > *relevant = vect_used_in_loop;
> >
> > /* changing memory. */
> > --- 2153,2160 ----
> > *live_p = false;
> >
> > /* cond stmt other than loop exit cond. */
> > ! if (is_ctrl_stmt (stmt)
> > ! && STMT_VINFO_TYPE (vinfo_for_stmt (stmt)) !=
> loop_exit_ctrl_vec_info_type)
> > *relevant = vect_used_in_loop;
> >
> > /* changing memory. */
> > *************** vect_stmt_relevant_p (tree stmt, loop_ve
> > *** 2180,2185 ****
> > --- 2191,2271 ----
> > }
> >
> >
> > + /*
> > + Function process_use.
> > +
> > + Inputs:
> > + - a USE in STMT in a loop represented by LOOP_VINFO
> > + - LIVE_P, RELEVANT - enum values to be set in the STMT_VINFO
> of the stmt
> > + that defined USE. This is dont by calling mark_relevant
andpassing it
> > + the WORKLIST (to add DEF_STMT to the WORKlist in case itis
> relevant).
> > +
> > + Outputs:
> > + Generally, LIVE_P and RELEVANT are used to define the liveness and
> > + relevance info of the DEF_STMT of this USE:
> > + STMT_VINFO_LIVE_P (DEF_STMT_info) <-- live_p
> > + STMT_VINFO_RELEVANT (DEF_STMT_info) <-- relevant
> > + Exceptions:
> > + - case 1: If USE is used only for address computations (e.g.
> array indexing),
> > + which does not need to be directly vectorized, then the
> liveness/relevance
> > + of the respective DEF_STMT is left unchanged.
> > + - case 2: If STMT is a reduction phi and DEF_STMT is a
> reduction stmt, we
> > + skip DEF_STMT cause it had already been processed.
> > +
> > + Return true if everyting is as expected. Return false otherwise.
*/
> > +
> > + static bool
> > + process_use (tree stmt, tree use, loop_vec_info loop_vinfo, bool
live_p,
> > + enum vect_relevant relevant, VEC(tree,heap) **worklist)
> > + {
> > + struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> > + stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
> > + stmt_vec_info dstmt_vinfo;
> > + basic_block def_bb;
> > + tree def, def_stmt;
> > + enum vect_def_type dt;
> > +
> > + /* case 1: we are only interested in uses that need to be
> vectorized. Uses
> > + that are used for address computation are not considered
> relevant. */
> > + if (!exist_non_indexing_operands_for_use_p (use, stmt))
> > + return true;
> > +
> > + if (!vect_is_simple_use (use, loop_vinfo, &def_stmt, &def, &dt))
> > + {
> > + if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > + fprintf (vect_dump, "not vectorized: unsupported use in
stmt.");
> > + return false;
> > + }
> > +
> > + if (!def_stmt || IS_EMPTY_STMT (def_stmt))
> > + return true;
> > +
> > + def_bb = bb_for_stmt (def_stmt);
> > + if (!flow_bb_inside_loop_p (loop, def_bb))
> > + return true;
> > +
> > + /* case 2: A reduction phi defining a reduction stmt
> (DEF_STMT). DEF_STMT
> > + must have already been processed, so we just check that
> everything is as
> > + expected, and we are done. */
> > + dstmt_vinfo = vinfo_for_stmt (def_stmt);
> > + if (TREE_CODE (stmt) == PHI_NODE
> > + && STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def
> > + && TREE_CODE (def_stmt) != PHI_NODE
> > + && STMT_VINFO_DEF_TYPE (dstmt_vinfo) == vect_reduction_def)
> > + {
> > + if (STMT_VINFO_IN_PATTERN_P (dstmt_vinfo))
> > + dstmt_vinfo = vinfo_for_stmt (STMT_VINFO_RELATED_STMT
(dstmt_vinfo));
> > + gcc_assert (STMT_VINFO_RELEVANT (dstmt_vinfo) <
> vect_used_by_reduction);
> > + gcc_assert (STMT_VINFO_LIVE_P (dstmt_vinfo)
> > + || STMT_VINFO_RELEVANT (dstmt_vinfo) > vect_unused_in_loop);
> > + return true;
> > + }
> > +
> > + vect_mark_relevant (worklist, def_stmt, relevant, live_p);
> > + return true;
> > + }
> > +
> > +
> > /* Function vect_mark_stmts_to_be_vectorized.
> >
> > Not all stmts in the loop need to be vectorized. For example:
> > *************** vect_mark_stmts_to_be_vectorized (loop_v
> > *** 2204,2220 ****
> > basic_block *bbs = LOOP_VINFO_BBS (loop_vinfo);
> > unsigned int nbbs = loop->num_nodes;
> > block_stmt_iterator si;
> > ! tree stmt, use;
> > stmt_ann_t ann;
> > - ssa_op_iter iter;
> > unsigned int i;
> > stmt_vec_info stmt_vinfo;
> > basic_block bb;
> > tree phi;
> > bool live_p;
> > enum vect_relevant relevant;
> > - tree def, def_stmt;
> > - enum vect_def_type dt;
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "=== vect_mark_stmts_to_be_vectorized ===");
> > --- 2290,2303 ----
> > basic_block *bbs = LOOP_VINFO_BBS (loop_vinfo);
> > unsigned int nbbs = loop->num_nodes;
> > block_stmt_iterator si;
> > ! tree stmt;
> > stmt_ann_t ann;
> > unsigned int i;
> > stmt_vec_info stmt_vinfo;
> > basic_block bb;
> > tree phi;
> > bool live_p;
> > enum vect_relevant relevant;
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "=== vect_mark_stmts_to_be_vectorized ===");
> > *************** vect_mark_stmts_to_be_vectorized (loop_v
> > *** 2222,2248 ****
> > worklist = VEC_alloc (tree, heap, 64);
> >
> > /* 1. Init worklist. */
> > -
> > - bb = loop->header;
> > - for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
> > - {
> > - if (vect_print_dump_info (REPORT_DETAILS))
> > - {
> > - fprintf (vect_dump, "init: phi relevant? ");
> > - print_generic_expr (vect_dump, phi, TDF_SLIM);
> > - }
> > -
> > - if (vect_stmt_relevant_p (phi, loop_vinfo, &relevant, &live_p))
> > - vect_mark_relevant (&worklist, phi, relevant, live_p);
> > - }
> > -
> > for (i = 0; i < nbbs; i++)
> > {
> > bb = bbs[i];
> > for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
> > {
> > stmt = bsi_stmt (si);
> > -
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > fprintf (vect_dump, "init: stmt relevant? ");
> > --- 2305,2327 ----
> > worklist = VEC_alloc (tree, heap, 64);
> >
> > /* 1. Init worklist. */
> > for (i = 0; i < nbbs; i++)
> > {
> > bb = bbs[i];
> > + for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
> > + {
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + {
> > + fprintf (vect_dump, "init: phi relevant? ");
> > + print_generic_expr (vect_dump, phi, TDF_SLIM);
> > + }
> > +
> > + if (vect_stmt_relevant_p (phi, loop_vinfo, &relevant, &live_p))
> > + vect_mark_relevant (&worklist, phi, relevant, live_p);
> > + }
> > for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
> > {
> > stmt = bsi_stmt (si);
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > fprintf (vect_dump, "init: stmt relevant? ");
> > *************** vect_mark_stmts_to_be_vectorized (loop_v
> > *** 2254,2355 ****
> > }
> > }
> >
> > -
> > /* 2. Process_worklist */
> > -
> > while (VEC_length (tree, worklist) > 0)
> > {
> > ! stmt = VEC_pop (tree, worklist);
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > fprintf (vect_dump, "worklist: examine stmt: ");
> > print_generic_expr (vect_dump, stmt, TDF_SLIM);
> > }
> >
> > ! /* Examine the USEs of STMT. For each ssa-name USE that is
defined
> > ! in the loop, mark the stmt that defines it (DEF_STMT) as
> > ! relevant/irrelevant and live/dead according to the liveness
and
> > ! relevance properties of STMT.
> > ! */
> > !
> > ! gcc_assert (TREE_CODE (stmt) != PHI_NODE);
> > !
> > ann = stmt_ann (stmt);
> > stmt_vinfo = vinfo_for_stmt (stmt);
> > -
> > relevant = STMT_VINFO_RELEVANT (stmt_vinfo);
> > live_p = STMT_VINFO_LIVE_P (stmt_vinfo);
> >
> > /* Generally, the liveness and relevance properties of STMT are
> > ! propagated to the DEF_STMTs of its USEs:
> > ! STMT_VINFO_LIVE_P (DEF_STMT_info) <-- live_p
> > ! STMT_VINFO_RELEVANT (DEF_STMT_info) <-- relevant
> > !
> > ! Exceptions:
> > !
> > ! (case 1)
> > ! If USE is used only for address computations (e.g.
> array indexing),
> > ! which does not need to be directly vectorized, then the
> > ! liveness/relevance of the respective DEF_STMT is
leftunchanged.
> > !
> > ! (case 2)
> > ! If STMT has been identified as defining a reduction
> variable, then
> > ! we want to set liveness/relevance as follows:
> > ! STMT_VINFO_LIVE_P (DEF_STMT_info) <-- false
> > ! STMT_VINFO_RELEVANT (DEF_STMT_info) <--
> vect_used_by_reduction
> > ! because even though STMT is classified as live
> (since it defines a
> > ! value that is used across loop iterations) and
> irrelevant (since it
> > ! is not used inside the loop), it will be
> vectorized, and therefore
> > ! the corresponding DEF_STMTs need to marked as relevant.
> > ! We distinguish between two kinds of relevant stmts -
> those that are
> > ! used by a reduction computation, and those that are (also)
used by
> > ! a regular computation. This allows us later on to identify
stmts
> > ! that are used solely by a reduction, and therefore the order
of
> > ! the results that they produce does not have to be kept.
> > ! */
> >
> > - /* case 2.2: */
> > if (STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def)
> > ! {
> > ! gcc_assert (relevant == vect_unused_in_loop && live_p);
> > relevant = vect_used_by_reduction;
> > ! live_p = false;
> > }
> >
> > ! i = 0;
> > ! FOR_EACH_SSA_TREE_OPERAND (use, stmt, iter, SSA_OP_USE)
> > {
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > - fprintf (vect_dump, "worklist: examine use %d: ", i++);
> > - print_generic_expr (vect_dump, use, TDF_SLIM);
> > - }
> > -
> > - /* case 1: we are only interested in uses that need to be
> vectorized.
> > - Uses that are used for address computation are not considered
> > - relevant.
> > - */
> > - if (!exist_non_indexing_operands_for_use_p (use, stmt))
> > - continue;
> > -
> > - if (!vect_is_simple_use (use, loop_vinfo, &def_stmt, &def, &dt))
> > - {
> > - if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > - fprintf (vect_dump, "not vectorized: unsupported use in
stmt.");
> > VEC_free (tree, heap, worklist);
> > return false;
> > ! }
> > !
> > ! if (!def_stmt || IS_EMPTY_STMT (def_stmt))
> > ! continue;
> > !
> > ! bb = bb_for_stmt (def_stmt);
> > ! if (!flow_bb_inside_loop_p (loop, bb))
> > ! continue;
> > ! vect_mark_relevant (&worklist, def_stmt, relevant, live_p);
> > }
> > ! } /* while worklist */
> >
> > VEC_free (tree, heap, worklist);
> > return true;
> > --- 2333,2414 ----
> > }
> > }
> >
> > /* 2. Process_worklist */
> > while (VEC_length (tree, worklist) > 0)
> > {
> > ! use_operand_p use_p;
> > ! ssa_op_iter iter;
> >
> > + stmt = VEC_pop (tree, worklist);
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > fprintf (vect_dump, "worklist: examine stmt: ");
> > print_generic_expr (vect_dump, stmt, TDF_SLIM);
> > }
> >
> > ! /* Examine the USEs of STMT. For each USE, mark the stmt
> that defines it
> > ! (DEF_STMT) as relevant/irrelevant and live/dead according to the
> > ! liveness and relevance properties of STMT. */
> > ann = stmt_ann (stmt);
> > stmt_vinfo = vinfo_for_stmt (stmt);
> > relevant = STMT_VINFO_RELEVANT (stmt_vinfo);
> > live_p = STMT_VINFO_LIVE_P (stmt_vinfo);
> >
> > /* Generally, the liveness and relevance properties of STMT are
> > ! propagated as is to the DEF_STMTs of its USEs:
> > ! live_p <-- STMT_VINFO_LIVE_P (STMT_VINFO)
> > ! relevant <-- STMT_VINFO_RELEVANT (STMT_VINFO)
> > !
> > ! One exception is when STMT has been identified as defining
areduction
> > ! variable; in this case we set the liveness/relevance as follows:
> > ! live_p = false
> > ! relevant = vect_used_by_reduction
> > ! This is because we distinguish between two kinds of relevant
stmts -
> > ! those that are used by a reduction computation, and those that
are
> > ! (also) used by a regular computation. This allows us later on to
> > ! identify stmts that are used solely by a reduction, and therefore
the
> > ! order of the results that they produce does not have to be kept.
> > !
> > ! Reduction phis are expected to be used by a reduction
> stmt; Other
> > ! reduction stmts are expected to be unused in the loop. These are
the
> > ! expected values of "relevant" for reduction phis/stmts in the
loop:
> > !
> > ! relevance: phi stmt
> > ! vect_unused_in_loop ok
> > ! vect_used_by_reduction ok
> > ! vect_used_in_loop */
> >
> > if (STMT_VINFO_DEF_TYPE (stmt_vinfo) == vect_reduction_def)
> > ! {
> > ! switch (relevant)
> > ! {
> > ! case vect_unused_in_loop:
> > ! gcc_assert (TREE_CODE (stmt) != PHI_NODE);
> > ! break;
> > ! case vect_used_by_reduction:
> > ! if (TREE_CODE (stmt) == PHI_NODE)
> > ! break;
> > ! case vect_used_in_loop:
> > ! default:
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "unsupported use of reduction.");
> > ! VEC_free (tree, heap, worklist);
> > ! return false;
> > ! }
> > relevant = vect_used_by_reduction;
> > ! live_p = false;
> > }
> >
> > ! FOR_EACH_PHI_OR_STMT_USE (use_p, stmt, iter, SSA_OP_USE)
> > {
> > ! tree op = USE_FROM_PTR (use_p);
> > ! if (!process_use (stmt, op, loop_vinfo, live_p, relevant,
&worklist))
> > {
> > VEC_free (tree, heap, worklist);
> > return false;
> > ! }
> > }
> > ! } /* while worklist */
> >
> > VEC_free (tree, heap, worklist);
> > return true;
> > *************** vect_analyze_loop_form (struct loop *loo
> > *** 2582,2591 ****
> > return false;
> > }
> >
> > ! loop_vinfo = new_loop_vec_info (loop);
> > ! LOOP_VINFO_NITERS (loop_vinfo) = number_of_iterations;
> > !
> > ! if (!LOOP_VINFO_NITERS_KNOWN_P (loop_vinfo))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > --- 2641,2647 ----
> > return false;
> > }
> >
> > ! if (!NITERS_KNOWN_P (number_of_iterations))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > {
> > *************** vect_analyze_loop_form (struct loop *loo
> > *** 2594,2608 ****
> > }
> > }
> > else
> > ! if (LOOP_VINFO_INT_NITERS (loop_vinfo) == 0)
> > {
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > fprintf (vect_dump, "not vectorized: number of iterations =
0.");
> > return NULL;
> > }
> >
> > ! LOOP_VINFO_EXIT_COND (loop_vinfo) = loop_cond;
> >
> > return loop_vinfo;
> > }
> >
> > --- 2650,2669 ----
> > }
> > }
> > else
> > ! if (TREE_INT_CST_LOW (number_of_iterations) == 0)
> > {
> > if (vect_print_dump_info (REPORT_UNVECTORIZED_LOOPS))
> > fprintf (vect_dump, "not vectorized: number of iterations =
0.");
> > return NULL;
> > }
> >
> > ! loop_vinfo = new_loop_vec_info (loop);
> > ! LOOP_VINFO_NITERS (loop_vinfo) = number_of_iterations;
> > !
> > ! STMT_VINFO_TYPE (vinfo_for_stmt (loop_cond)) =
> loop_exit_ctrl_vec_info_type;
> >
> > + gcc_assert (!loop->aux);
> > + loop->aux = loop_vinfo;
> > return loop_vinfo;
> > }
> >
> > Index: common.opt
> > ===================================================================
> > *** common.opt (revision 123103)
> > --- common.opt (working copy)
> > *************** funroll-all-loops
> > *** 1070,1075 ****
> > --- 1070,1079 ----
> > Common Report Var(flag_unroll_all_loops) Optimization
> > Perform loop unrolling for all loops
> >
> > + ftree-reassoc
> > + Common Report Var(flag_tree_reassoc) Init(1) Optimization
> > + Perform reassociation on trees
> > +
> > ; Nonzero means that loop optimizer may assume that the
> induction variables
> > ; that control loops do not overflow and that the loops with
nontrivial
> > ; exit condition are not infinite
> > Index: tree-vect-patterns.c
> > ===================================================================
> > *** tree-vect-patterns.c (revision 123103)
> > --- tree-vect-patterns.c (working copy)
> > *************** widened_name_p (tree name, tree use_stmt
> > *** 109,118 ****
> > if (!vect_is_simple_use (oprnd0, loop_vinfo, &dummy, &dummy, &dt))
> > return false;
> >
> > - if (dt != vect_invariant_def && dt != vect_constant_def
> > - && dt != vect_loop_def)
> > - return false;
> > -
> > return true;
> > }
> >
> > --- 109,114 ----
> > Index: tree-vect-transform.c
> > ===================================================================
> > *** tree-vect-transform.c (revision 123103)
> > --- tree-vect-transform.c (working copy)
> > *************** vect_init_vector (tree stmt, tree vector
> > *** 514,520 ****
> > /* Function get_initial_def_for_induction
> >
> > Input:
> > - STMT - a stmt that performs an induction operation in the loop.
> > IV_PHI - the initial value of the induction variable
> >
> > Output:
> > --- 514,519 ----
> > *************** vect_init_vector (tree stmt, tree vector
> > *** 524,532 ****
> > [X, X + S, X + 2*S, X + 3*S]. */
> >
> > static tree
> > ! get_initial_def_for_induction (tree stmt, tree iv_phi)
> > {
> > ! stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
> > loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_vinfo);
> > struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> > tree scalar_type = TREE_TYPE (iv_phi);
> > --- 523,531 ----
> > [X, X + S, X + 2*S, X + 3*S]. */
> >
> > static tree
> > ! get_initial_def_for_induction (tree iv_phi)
> > {
> > ! stmt_vec_info stmt_vinfo = vinfo_for_stmt (iv_phi);
> > loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_vinfo);
> > struct loop *loop = LOOP_VINFO_LOOP (loop_vinfo);
> > tree scalar_type = TREE_TYPE (iv_phi);
> > *************** get_initial_def_for_induction (tree stmt
> > *** 549,575 ****
> > tree expr;
> > stmt_vec_info phi_info = vinfo_for_stmt (iv_phi);
> > tree stmts;
> >
> > gcc_assert (phi_info);
> >
> > ! if (STMT_VINFO_VEC_STMT (phi_info))
> > {
> > ! induction_phi = STMT_VINFO_VEC_STMT (phi_info);
> > ! gcc_assert (TREE_CODE (induction_phi) == PHI_NODE);
> > !
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > ! {
> > ! fprintf (vect_dump, "induction already vectorized:");
> > ! print_generic_expr (vect_dump, iv_phi, TDF_SLIM);
> > ! fprintf (vect_dump, "\n");
> > ! print_generic_expr (vect_dump, induction_phi, TDF_SLIM);
> > ! }
> > !
> > ! return PHI_RESULT (induction_phi);
> > }
> >
> > - gcc_assert (ncopies >= 1);
> > -
> > access_fn = analyze_scalar_evolution (loop, PHI_RESULT (iv_phi));
> > gcc_assert (access_fn);
> > ok = vect_is_simple_iv_evolution (loop->num, access_fn,
> &init_expr, &step_expr);
> > --- 548,568 ----
> > tree expr;
> > stmt_vec_info phi_info = vinfo_for_stmt (iv_phi);
> > tree stmts;
> > + tree stmt = NULL_TREE;
> > + block_stmt_iterator si;
> > + basic_block bb = bb_for_stmt (iv_phi);
> >
> > gcc_assert (phi_info);
> > + gcc_assert (ncopies >= 1);
> >
> > ! /* Find the first insertion point in the BB. */
> > ! for (si = bsi_start (bb); !bsi_end_p (si); bsi_next (&si))
> > {
> > ! stmt = bsi_stmt (si);
> > ! if (TREE_CODE (stmt) != LABEL_EXPR)
> > ! break;
> > }
> >
> > access_fn = analyze_scalar_evolution (loop, PHI_RESULT (iv_phi));
> > gcc_assert (access_fn);
> > ok = vect_is_simple_iv_evolution (loop->num, access_fn,
> &init_expr, &step_expr);
> > *************** vect_get_vec_def_for_operand (tree op, t
> > *** 833,839 ****
> > gcc_assert (TREE_CODE (def_stmt) == PHI_NODE);
> >
> > /* Get the def before the loop */
> > ! return get_initial_def_for_induction (stmt, def_stmt);
> > }
> >
> > default:
> > --- 826,832 ----
> > gcc_assert (TREE_CODE (def_stmt) == PHI_NODE);
> >
> > /* Get the def before the loop */
> > ! return get_initial_def_for_induction (def_stmt);
> > }
> >
> > default:
> > *************** vect_finish_stmt_generation (tree stmt,
> > *** 951,958 ****
> > }
> >
> >
> > - #define ADJUST_IN_EPILOG 1
> > -
> > /* Function get_initial_def_for_reduction
> >
> > Input:
> > --- 944,949 ----
> > *************** vect_finish_stmt_generation (tree stmt,
> > *** 960,977 ****
> > INIT_VAL - the initial value of the reduction variable
> >
> > Output:
> > ! SCALAR_DEF - a tree that holds a value to be added to the final
result
> > ! of the reduction (used for "ADJUST_IN_EPILOG" - see below).
> > Return a vector variable, initialized according to the
> operation that STMT
> > ! performs. This vector will be used as the initial value of the
> > ! vector of partial results.
> >
> > ! Option1 ("ADJUST_IN_EPILOG"): Initialize the vector as follows:
> > add: [0,0,...,0,0]
> > mult: [1,1,...,1,1]
> > min/max: [init_val,init_val,..,init_val,init_val]
> > bit and/or: [init_val,init_val,..,init_val,init_val]
> > ! and when necessary (e.g. add/mult case) let the caller know
> > that it needs to adjust the result by init_val.
> >
> > Option2: Initialize the vector as follows:
> > --- 951,968 ----
> > INIT_VAL - the initial value of the reduction variable
> >
> > Output:
> > ! ADJUSTMENT_DEF - a tree that holds a value to be added to the
> final result
> > ! of the reduction (used for adjusting the epilog - see below).
> > Return a vector variable, initialized according to the
> operation that STMT
> > ! performs. This vector will be used as the initial value of
the
> > ! vector of partial results.
> >
> > ! Option1 (adjust in epilog): Initialize the vector as follows:
> > add: [0,0,...,0,0]
> > mult: [1,1,...,1,1]
> > min/max: [init_val,init_val,..,init_val,init_val]
> > bit and/or: [init_val,init_val,..,init_val,init_val]
> > ! and when necessary (e.g. add/mult case) let the caller know
> > that it needs to adjust the result by init_val.
> >
> > Option2: Initialize the vector as follows:
> > *************** vect_finish_stmt_generation (tree stmt,
> > *** 992,1075 ****
> > or [0,0,0,0] and let the caller know that it needs to adjust
> > the result at the end by 'init_val'.
> >
> > ! FORNOW: We use the "ADJUST_IN_EPILOG" scheme.
> > ! TODO: Use some cost-model to estimate which scheme is more
profitable.
> > ! */
> >
> > static tree
> > ! get_initial_def_for_reduction (tree stmt, tree init_val, tree
*scalar_def)
> > {
> > stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
> > tree vectype = STMT_VINFO_VECTYPE (stmt_vinfo);
> > int nunits = TYPE_VECTOR_SUBPARTS (vectype);
> > - int nelements;
> > enum tree_code code = TREE_CODE (GIMPLE_STMT_OPERAND (stmt, 1));
> > tree type = TREE_TYPE (init_val);
> > ! tree def;
> > ! tree vec, t = NULL_TREE;
> > ! bool need_epilog_adjust;
> > int i;
> > tree vector_type;
> >
> > gcc_assert (INTEGRAL_TYPE_P (type) || SCALAR_FLOAT_TYPE_P (type));
> >
> > switch (code)
> > {
> > case WIDEN_SUM_EXPR:
> > case DOT_PROD_EXPR:
> > case PLUS_EXPR:
> > if (INTEGRAL_TYPE_P (type))
> > ! def = build_int_cst (type, 0);
> > else
> > ! def = build_real (type, dconst0);
> > !
> > ! #ifdef ADJUST_IN_EPILOG
> > ! /* All the 'nunits' elements are set to 0. The final result will
be
> > ! adjusted by 'init_val' at the loop epilog. */
> > ! nelements = nunits;
> > ! need_epilog_adjust = true;
> > ! #else
> > ! /* 'nunits - 1' elements are set to 0; The last element is set to
> > ! 'init_val'. No further adjustments at the epilog are needed.
*/
> > ! nelements = nunits - 1;
> > ! need_epilog_adjust = false;
> > ! #endif
> > break;
> >
> > case MIN_EXPR:
> > case MAX_EXPR:
> > ! def = init_val;
> > ! nelements = nunits;
> > ! need_epilog_adjust = false;
> > break;
> >
> > default:
> > gcc_unreachable ();
> > }
> >
> > ! for (i = nelements - 1; i >= 0; --i)
> > ! t = tree_cons (NULL_TREE, def, t);
> > !
> > ! if (nelements == nunits - 1)
> > ! {
> > ! /* Set the last element of the vector. */
> > ! t = tree_cons (NULL_TREE, init_val, t);
> > ! nelements += 1;
> > ! }
> > ! gcc_assert (nelements == nunits);
> > !
> > ! vector_type = get_vectype_for_scalar_type (TREE_TYPE (def));
> > ! if (TREE_CODE (init_val) == INTEGER_CST || TREE_CODE
> (init_val) == REAL_CST)
> > ! vec = build_vector (vector_type, t);
> > ! else
> > ! vec = build_constructor_from_list (vector_type, t);
> > !
> > ! if (!need_epilog_adjust)
> > ! *scalar_def = NULL_TREE;
> > ! else
> > ! *scalar_def = init_val;
> > !
> > ! return vect_init_vector (stmt, vec, vector_type);
> > }
> >
> >
> > --- 983,1038 ----
> > or [0,0,0,0] and let the caller know that it needs to adjust
> > the result at the end by 'init_val'.
> >
> > ! FORNOW, we are using the 'adjust in epilog' scheme, because
> this way the
> > ! initialization vector is simpler (same element in all entries).
> > ! A cost model should help decide between these two schemes. */
> >
> > static tree
> > ! get_initial_def_for_reduction (tree stmt, tree init_val, tree
> *adjustment_def)
> > {
> > stmt_vec_info stmt_vinfo = vinfo_for_stmt (stmt);
> > tree vectype = STMT_VINFO_VECTYPE (stmt_vinfo);
> > int nunits = TYPE_VECTOR_SUBPARTS (vectype);
> > enum tree_code code = TREE_CODE (GIMPLE_STMT_OPERAND (stmt, 1));
> > tree type = TREE_TYPE (init_val);
> > ! tree vecdef;
> > ! tree def_for_init;
> > ! tree init_def;
> > ! tree t = NULL_TREE;
> > int i;
> > tree vector_type;
> >
> > gcc_assert (INTEGRAL_TYPE_P (type) || SCALAR_FLOAT_TYPE_P (type));
> > + vecdef = vect_get_vec_def_for_operand (init_val, stmt, NULL);
> >
> > switch (code)
> > {
> > case WIDEN_SUM_EXPR:
> > case DOT_PROD_EXPR:
> > case PLUS_EXPR:
> > + *adjustment_def = init_val;
> > + /* Create a vector of zeros for init_def. */
> > if (INTEGRAL_TYPE_P (type))
> > ! def_for_init = build_int_cst (type, 0);
> > else
> > ! def_for_init = build_real (type, dconst0);
> > ! for (i = nunits - 1; i >= 0; --i)
> > ! t = tree_cons (NULL_TREE, def_for_init, t);
> > ! vector_type = get_vectype_for_scalar_type (TREE_TYPE
(def_for_init));
> > ! init_def = build_vector (vector_type, t);
> > break;
> >
> > case MIN_EXPR:
> > case MAX_EXPR:
> > ! *adjustment_def = NULL_TREE;
> > ! init_def = vecdef;
> > break;
> >
> > default:
> > gcc_unreachable ();
> > }
> >
> > ! return init_def;
> > }
> >
> >
> > *************** vectorizable_call (tree stmt, block_stmt
> > *** 1816,1821 ****
> > --- 1779,1798 ----
> > int ncopies, j, nargs;
> > call_expr_arg_iterator iter;
> >
> > + if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > + return false;
> > +
> > + if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > + return false;
> > +
> > + /* FORNOW: not yet supported. */
> > + if (STMT_VINFO_LIVE_P (stmt_info))
> > + {
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + fprintf (vect_dump, "value used after loop.");
> > + return false;
> > + }
> > +
> > /* Is STMT a vectorizable call? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> > *************** vectorizable_conversion (tree stmt, bloc
> > *** 1994,2000 ****
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> >
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > --- 1971,1978 ----
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> >
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > *************** vectorizable_assignment (tree stmt, bloc
> > *** 2134,2145 ****
> > if (ncopies > 1)
> > return false; /* FORNOW */
> >
> > - /* Is vectorizable assignment? */
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > --- 2112,2132 ----
> > if (ncopies > 1)
> > return false; /* FORNOW */
> >
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> > !
> > ! /* FORNOW: not yet supported. */
> > ! if (STMT_VINFO_LIVE_P (stmt_info))
> > ! {
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "value used after loop.");
> > ! return false;
> > ! }
> >
> > + /* Is vectorizable assignment? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > *************** vect_min_worthwhile_factor (enum tree_co
> > *** 2209,2214 ****
> > --- 2196,2254 ----
> > }
> >
> >
> > + /* Function vectorizable_induction
> > +
> > + Check if PHI performs an induction computation that can be
vectorized.
> > + If VEC_STMT is also passed, vectorize the induction PHI:
> create a vectorized
> > + phi to replace it, put it in VEC_STMT, and add it to the same
> basic block.
> > + Return FALSE if not a vectorizable STMT, TRUE otherwise. */
> > +
> > + bool
> > + vectorizable_induction (tree phi, block_stmt_iterator *bsi
> ATTRIBUTE_UNUSED,
> > + tree *vec_stmt)
> > + {
> > + stmt_vec_info stmt_info = vinfo_for_stmt (phi);
> > + tree vectype = STMT_VINFO_VECTYPE (stmt_info);
> > + loop_vec_info loop_vinfo = STMT_VINFO_LOOP_VINFO (stmt_info);
> > + int nunits = TYPE_VECTOR_SUBPARTS (vectype);
> > + int ncopies = LOOP_VINFO_VECT_FACTOR (loop_vinfo) / nunits;
> > + tree vec_def;
> > +
> > + gcc_assert (ncopies >= 1);
> > +
> > + if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > + return false;
> > +
> > + gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def);
> > +
> > + if (STMT_VINFO_LIVE_P (stmt_info))
> > + {
> > + /* FORNOW: not yet supported. */
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + fprintf (vect_dump, "value used after loop.");
> > + return false;
> > + }
> > +
> > + if (TREE_CODE (phi) != PHI_NODE)
> > + return false;
> > +
> > + if (!vec_stmt) /* transformation not required. */
> > + {
> > + STMT_VINFO_TYPE (stmt_info) = induc_vec_info_type;
> > + return true;
> > + }
> > +
> > + /** Transform. **/
> > +
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + fprintf (vect_dump, "transform induction phi.");
> > +
> > + vec_def = get_initial_def_for_induction (phi);
> > + *vec_stmt = SSA_NAME_DEF_STMT (vec_def);
> > + return true;
> > + }
> > +
> > +
> > /* Function vectorizable_operation.
> >
> > Check if STMT performs a binary or unary operation that can
> be vectorized.
> > *************** vectorizable_operation (tree stmt, block
> > *** 2246,2265 ****
> >
> > gcc_assert (ncopies >= 1);
> >
> > - /* Is STMT a vectorizable binary/unary operation? */
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> >
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > - /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > --- 2286,2306 ----
> >
> > gcc_assert (ncopies >= 1);
> >
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> >
> > + /* FORNOW: not yet supported. */
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > + /* Is STMT a vectorizable binary/unary operation? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > *************** vectorizable_type_demotion (tree stmt, b
> > *** 2514,2534 ****
> > optab optab;
> > enum machine_mode vec_mode;
> >
> > - /* Is STMT a vectorizable type-demotion operation? */
> > -
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> > !
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> > !
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > - /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > --- 2555,2575 ----
> > optab optab;
> > enum machine_mode vec_mode;
> >
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> > !
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> > !
> > ! /* FORNOW: not yet supported. */
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > + /* Is STMT a vectorizable type-demotion operation? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > *************** vectorizable_type_promotion (tree stmt,
> > *** 2723,2743 ****
> > int j;
> > tree vectype_in;
> >
> > - /* Is STMT a vectorizable type-promotion operation? */
> > -
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> >
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > - /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > --- 2764,2784 ----
> > int j;
> > tree vectype_in;
> >
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> >
> > + /* FORNOW: not yet supported. */
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > + /* Is STMT a vectorizable type-promotion operation? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > *************** vectorizable_store (tree stmt, block_stm
> > *** 3064,3069 ****
> > --- 3105,3123 ----
> > VEC(tree,heap) *dr_chain = NULL, *oprnds = NULL, *result_chain =
NULL;
> > gcc_assert (ncopies >= 1);
> >
> > + if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > + return false;
> > +
> > + if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > + return false;
> > +
> > + if (STMT_VINFO_LIVE_P (stmt_info))
> > + {
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + fprintf (vect_dump, "value used after loop.");
> > + return false;
> > + }
> > +
> > /* Is vectorizable store? */
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > *************** vectorizable_load (tree stmt, block_stmt
> > *** 3710,3729 ****
> > bool strided_load = false;
> > tree first_stmt;
> >
> > - /* Is vectorizable load? */
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> >
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > - /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > --- 3764,3784 ----
> > bool strided_load = false;
> > tree first_stmt;
> >
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> >
> > + /* FORNOW: not yet supported. */
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > + /* Is vectorizable load? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > *************** vectorizable_live_operation (tree stmt,
> > *** 4010,4016 ****
> > tree def, def_stmt;
> > enum vect_def_type dt;
> >
> > ! if (!STMT_VINFO_LIVE_P (stmt_info))
> > return false;
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > --- 4065,4073 ----
> > tree def, def_stmt;
> > enum vect_def_type dt;
> >
> > ! gcc_assert (STMT_VINFO_LIVE_P (stmt_info));
> > !
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_reduction_def)
> > return false;
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > *************** vectorizable_condition (tree stmt, block
> > *** 4123,4138 ****
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! gcc_assert (STMT_VINFO_DEF_TYPE (stmt_info) == vect_loop_def);
> >
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > - /* FORNOW: not yet supported. */
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > --- 4180,4197 ----
> > if (!STMT_VINFO_RELEVANT_P (stmt_info))
> > return false;
> >
> > ! if (STMT_VINFO_DEF_TYPE (stmt_info) != vect_loop_def)
> > ! return false;
> >
> > + /* FORNOW: not yet supported. */
> > if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > if (vect_print_dump_info (REPORT_DETAILS))
> > fprintf (vect_dump, "value used after loop.");
> > return false;
> > }
> >
> > + /* Is vectorizable conditional operation? */
> > if (TREE_CODE (stmt) != GIMPLE_MODIFY_STMT)
> > return false;
> >
> > *************** vect_transform_stmt (tree stmt, block_st
> > *** 4225,4334 ****
> > tree orig_stmt_in_pattern;
> > bool done;
> >
> > ! if (STMT_VINFO_RELEVANT_P (stmt_info))
> > {
> > ! switch (STMT_VINFO_TYPE (stmt_info))
> > ! {
> > ! case type_demotion_vec_info_type:
> > ! done = vectorizable_type_demotion (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> >
> > ! case type_promotion_vec_info_type:
> > ! done = vectorizable_type_promotion (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case type_conversion_vec_info_type:
> > ! done = vectorizable_conversion (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case op_vec_info_type:
> > ! done = vectorizable_operation (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case assignment_vec_info_type:
> > ! done = vectorizable_assignment (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case load_vec_info_type:
> > ! done = vectorizable_load (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case store_vec_info_type:
> > ! done = vectorizable_store (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! if (DR_GROUP_FIRST_DR (stmt_info))
> > ! {
> > ! /* In case of interleaving, the whole chain is vectorized when
the
> > ! last store in the chain is reached. Store stmts before the
last
> > ! one are skipped, and there vec_stmt_info shouldn't be freed
> > ! meanwhile. */
> > ! *strided_store = true;
> > ! if (STMT_VINFO_VEC_STMT (stmt_info))
> > ! is_store = true;
> > }
> > ! else
> > ! is_store = true;
> > ! break;
> > !
> > ! case condition_vec_info_type:
> > ! done = vectorizable_condition (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> >
> > ! case call_vec_info_type:
> > ! done = vectorizable_call (stmt, bsi, &vec_stmt);
> > ! break;
> >
> > ! default:
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "stmt not supported.");
> > ! gcc_unreachable ();
> > ! }
> > !
> > ! gcc_assert (vec_stmt || *strided_store);
> > ! if (vec_stmt)
> > {
> > ! STMT_VINFO_VEC_STMT (stmt_info) = vec_stmt;
> > ! orig_stmt_in_pattern = STMT_VINFO_RELATED_STMT (stmt_info);
> > ! if (orig_stmt_in_pattern)
> > ! {
> > ! stmt_vec_info stmt_vinfo = vinfo_for_stmt
(orig_stmt_in_pattern);
> > ! if (STMT_VINFO_IN_PATTERN_P (stmt_vinfo))
> > ! {
> > ! gcc_assert (STMT_VINFO_RELATED_STMT (stmt_vinfo) == stmt);
> > !
> > ! /* STMT was inserted by the vectorizer to replace a
> > ! computation idiom. ORIG_STMT_IN_PATTERN is a stmt in the
> > ! original sequence that computed this idiom. We need to
> > ! record a pointer to VEC_STMT in the stmt_info of
> > ! ORIG_STMT_IN_PATTERN. See more details in the
> > ! documentation of vect_pattern_recog. */
> > !
> > ! STMT_VINFO_VEC_STMT (stmt_vinfo) = vec_stmt;
> > ! }
> > ! }
> > }
> > }
> >
> > ! if (STMT_VINFO_LIVE_P (stmt_info))
> > {
> > ! switch (STMT_VINFO_TYPE (stmt_info))
> > ! {
> > ! case reduc_vec_info_type:
> > ! done = vectorizable_reduction (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> >
> > ! default:
> > ! done = vectorizable_live_operation (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! }
> > }
> >
> > return is_store;
> > --- 4284,4391 ----
> > tree orig_stmt_in_pattern;
> > bool done;
> >
> > ! switch (STMT_VINFO_TYPE (stmt_info))
> > {
> > ! case type_demotion_vec_info_type:
> > ! done = vectorizable_type_demotion (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> >
> > ! case type_promotion_vec_info_type:
> > ! done = vectorizable_type_promotion (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case type_conversion_vec_info_type:
> > ! done = vectorizable_conversion (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case induc_vec_info_type:
> > ! done = vectorizable_induction (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case op_vec_info_type:
> > ! done = vectorizable_operation (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case assignment_vec_info_type:
> > ! done = vectorizable_assignment (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case load_vec_info_type:
> > ! done = vectorizable_load (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case store_vec_info_type:
> > ! done = vectorizable_store (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! if (DR_GROUP_FIRST_DR (stmt_info))
> > ! {
> > ! /* In case of interleaving, the whole chain is vectorized when
the
> > ! last store in the chain is reached. Store stmts before the
last
> > ! one are skipped, and there vec_stmt_info shouldn't be freed
> > ! meanwhile. */
> > ! *strided_store = true;
> > ! if (STMT_VINFO_VEC_STMT (stmt_info))
> > ! is_store = true;
> > }
> > ! else
> > ! is_store = true;
> > ! break;
> >
> > ! case condition_vec_info_type:
> > ! done = vectorizable_condition (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> > !
> > ! case call_vec_info_type:
> > ! done = vectorizable_call (stmt, bsi, &vec_stmt);
> > ! break;
> > !
> > ! case reduc_vec_info_type:
> > ! done = vectorizable_reduction (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! break;
> >
> > ! default:
> > ! if (!STMT_VINFO_LIVE_P (stmt_info))
> > {
> > ! if (vect_print_dump_info (REPORT_DETAILS))
> > ! fprintf (vect_dump, "stmt not supported.");
> > ! gcc_unreachable ();
> > }
> > }
> >
> > ! if (STMT_VINFO_LIVE_P (stmt_info)
> > ! && STMT_VINFO_TYPE (stmt_info) != reduc_vec_info_type)
> > {
> > ! done = vectorizable_live_operation (stmt, bsi, &vec_stmt);
> > ! gcc_assert (done);
> > ! }
> >
> > ! if (vec_stmt)
> > ! {
> > ! STMT_VINFO_VEC_STMT (stmt_info) = vec_stmt;
> > ! orig_stmt_in_pattern = STMT_VINFO_RELATED_STMT (stmt_info);
> > ! if (orig_stmt_in_pattern)
> > ! {
> > ! stmt_vec_info stmt_vinfo = vinfo_for_stmt
(orig_stmt_in_pattern);
> > ! /* STMT was inserted by the vectorizer to replace a
> computation idiom.
> > ! ORIG_STMT_IN_PATTERN is a stmt in the original sequence that
> > ! computed this idiom. We need to record a pointer to VEC_STMT
in
> > ! the stmt_info of ORIG_STMT_IN_PATTERN. See more details in
the
> > ! documentation of vect_pattern_recog. */
> > ! if (STMT_VINFO_IN_PATTERN_P (stmt_vinfo))
> > ! {
> > ! gcc_assert (STMT_VINFO_RELATED_STMT (stmt_vinfo) == stmt);
> > ! STMT_VINFO_VEC_STMT (stmt_vinfo) = vec_stmt;
> > ! }
> > ! }
> > }
> >
> > return is_store;
> > *************** vect_transform_loop (loop_vec_info loop_
> > *** 5156,5166 ****
> > for (i = 0; i < nbbs; i++)
> > {
> > basic_block bb = bbs[i];
> >
> > for (si = bsi_start (bb); !bsi_end_p (si);)
> > {
> > tree stmt = bsi_stmt (si);
> > - stmt_vec_info stmt_info;
> > bool is_store;
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > --- 5213,5251 ----
> > for (i = 0; i < nbbs; i++)
> > {
> > basic_block bb = bbs[i];
> > + stmt_vec_info stmt_info;
> > + tree phi;
> > +
> > + for (phi = phi_nodes (bb); phi; phi = PHI_CHAIN (phi))
> > + {
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + {
> > + fprintf (vect_dump, "------>vectorizing phi: ");
> > + print_generic_expr (vect_dump, phi, TDF_SLIM);
> > + }
> > + stmt_info = vinfo_for_stmt (phi);
> > + if (!stmt_info)
> > + continue;
> > + if (!STMT_VINFO_RELEVANT_P (stmt_info)
> > + && !STMT_VINFO_LIVE_P (stmt_info))
> > + continue;
> > +
> > + if ((TYPE_VECTOR_SUBPARTS (STMT_VINFO_VECTYPE (stmt_info))
> > + != (unsigned HOST_WIDE_INT) vectorization_factor)
> > + && vect_print_dump_info (REPORT_DETAILS))
> > + fprintf (vect_dump, "multiple-types.");
> > +
> > + if (STMT_VINFO_DEF_TYPE (stmt_info) == vect_induction_def)
> > + {
> > + if (vect_print_dump_info (REPORT_DETAILS))
> > + fprintf (vect_dump, "transform phi.");
> > + vect_transform_stmt (phi, NULL, NULL);
> > + }
> > + }
> >
> > for (si = bsi_start (bb); !bsi_end_p (si);)
> > {
> > tree stmt = bsi_stmt (si);
> > bool is_store;
> >
> > if (vect_print_dump_info (REPORT_DETAILS))
> > *************** vect_transform_loop (loop_vec_info loop_
> > *** 5177,5182 ****
> > --- 5262,5268 ----
> > continue;
> > }
> >
> > + gcc_assert (STMT_VINFO_VECTYPE (stmt_info));
> > if ((TYPE_VECTOR_SUBPARTS (STMT_VINFO_VECTYPE (stmt_info))
> > != (unsigned HOST_WIDE_INT) vectorization_factor)
> > && vect_print_dump_info (REPORT_DETAILS))
> > Index: tree-ssa-reassoc.c
> > ===================================================================
> > *** tree-ssa-reassoc.c (revision 123103)
> > --- tree-ssa-reassoc.c (working copy)
> > *************** execute_reassoc (void)
> > *** 1476,1485 ****
> > return 0;
> > }
> >
> > struct tree_opt_pass pass_reassoc =
> > {
> > "reassoc", /* name */
> > ! NULL, /* gate */
> > execute_reassoc, /* execute */
> > NULL, /* sub */
> > NULL, /* next */
> > --- 1476,1491 ----
> > return 0;
> > }
> >
> > + static bool
> > + gate_reassoc (void)
> > + {
> > + return flag_tree_reassoc != 0;
> > + }
> > +
> > struct tree_opt_pass pass_reassoc =
> > {
> > "reassoc", /* name */
> > ! gate_reassoc, /* gate */
> > execute_reassoc, /* execute */
> > NULL, /* sub */
> > NULL, /* next */
>
More information about the Gcc-patches
mailing list