This is the mail archive of the
gcc-patches@gcc.gnu.org
mailing list for the GCC project.
Re: [6/6] Link imm uses for pattern stmts
- From: Richard Biener <richard dot guenther at gmail dot com>
- To: GCC Patches <gcc-patches at gcc dot gnu dot org>, Richard Sandiford <richard dot sandiford at arm dot com>, Michael Matz <matz at suse dot de>
- Date: Wed, 29 Aug 2018 09:42:41 +0200
- Subject: Re: [6/6] Link imm uses for pattern stmts
- References: <87tvnerb5m.fsf@arm.com> <8736uyraw0.fsf@arm.com>
On Tue, Aug 28, 2018 at 1:25 PM Richard Sandiford
<richard.sandiford@arm.com> wrote:
>
> One of the warts of the vectoriser IR is that it doesn't link SSA name
> uses for pattern statements, leading to complicated special cases in
> vect_mark_stmts_to_be_vectorized and (especially) vect_detect_hybrid_slp.
> It also makes it harder to check how an SSA name is used after pattern
> replacement (something I need for a later patch).
>
> This patch adds a mode in which tree-ssa-operands.c can update
> statements in the same non-invasive way as for debug statements.
> It then uses this mode to update pattern statements when adding
> them to a vec_basic_block, so that pattern statements become
> even more like statements that existed from the outset.
Without reviewing in detail yet I think putting them in the same
ballpark as debug stmts or documenting they have no effect on
code generation is misleading - this is because immediate use
handling _does_ affect code generation unless you mark those
stmts in some way and handle them like debug-uses in all of
the immediate use routines. For example match-and-simplify
foldings guarded by :s will see the transparent stmts as uses.
So the real effect is to not add virtual operands or marking things
addressable? IMHO rather than adding a transparent_p flag
we eventually should allow passing down a oep flag to update_stmt?
Do vector pattern stmts cause any of the issues? I don't remember
seeing things that would cause TREE_ADDRESSABLE. I know
we have patterns for stores so we'd get extra VDEFs.
IIRC we historically didn't add pattern stmts to the IL simply because
pattern recognition was supposed to be analysis phase and building
and eventually tearing down pattern stmts was thought as to be too
expensive. That is also because we'd need to copy the loop to
retain a copy w/o those stmts.
So - would update_stmt (stmt, opf_no_vops|opf_non_addressable)
work? I suppose we'd need to pass down a second sticky_flags
arg.
I know Micha has some cleanup in the tree-ssa-operands.c area
sitting for too long on his disk so maybe he can at least share
if he got all that opf_non_addressable & opf_not_non_addressable
stuff simplified somehow...
Richard.
> 2018-08-28 Richard Sandiford <richard.sandiford@arm.com>
>
> gcc/
> * tree-ssa-operands.h (update_stmt_operands): Add a transparent_p
> argument.
> * tree-ssa-operands.c (opf_transparent, opf_sticky): New macros.
> (get_mem_ref_operands, get_tmr_operands): Preserve opf_sticky
> rather than just opf_no_vops.
> (get_expr_operands): Preserve opf_sticky bits in the use flags.
> Assert that opf_no_vops and opf_transparent are already set
> for the debug statements. Use opf_transparent rather than
> is_gimple_debug when deciding whether to mark something as
> having its address taken.
> (parse_ssa_operands): Add a transparent_p argument. Set the
> opf_no_vops and opf_transparent flags when the argument is true,
> or when dealing with debug statements. Check opf_no_vops before
> adding vuses and vdefs.
> (build_ssa_operands): Add a transparent_p argument and pass it to
> parse_ssa_operands.
> (verify_ssa_operands): Update call to parse_ssa_operands.
> (update_stmt_operands): Add a transparent_p argument and pass it to
> build_ssa_operands.
> * gimple-ssa.h (update_stmt, update_stmt_if_modified)
> (update_stmt_fn): Add an optional transparent_p parameter and
> update call to update_stmt_operands.
> * tree-vect-slp.c (vect_detect_hybrid_slp_1): Delete.
> (vect_detect_hybrid_slp_2): Likewise.
> (vect_detect_hybrid_slp): Don't treat pattern statements specially.
> * tree-vect-stmts.c (vect_mark_stmts_to_be_vectorized): Likewise.
> (vect_remove_dead_scalar_stmts): Remove pattern statements from
> the containing vec_info.
> * tree-vectorizer.h (vec_info::add_pattern_stmt_to_block): Declare.
> * tree-vectorizer.c (vec_basic_block::add_to_end)
> (vec_basic_block::add_before): Call add_pattern_stmt_to_block.
> (vec_basic_block::remove, vec_info::remove_stmt): Call
> remove_pattern_stmt_from_block.
> (vec_basic_block::add_pattern_stmt_to_block): New function.
> (remove_pattern_stmt_from_block): Likewise.
> (vec_info::free_stmt_vec_info): Handle pattern statements.
> (vec_info::lookup_single_use): Accept pattern statements
> as well as original statements. Ignore uses in statements
> that have been replaced by a pattern statement.
> * tree-vect-patterns.c (vect_init_pattern_stmt): Don't call
> gimple_set_bb.
> (vect_look_through_possible_promotion): Use vinfo->lookup_single_use
> instead of has_single_use. Track single uses for pattern statements
> too.
>
> Index: gcc/tree-ssa-operands.h
> ===================================================================
> --- gcc/tree-ssa-operands.h 2018-05-02 08:37:32.405761509 +0100
> +++ gcc/tree-ssa-operands.h 2018-08-28 12:05:19.262917177 +0100
> @@ -94,7 +94,7 @@ extern void init_ssa_operands (struct fu
> extern void fini_ssa_operands (struct function *);
> extern bool verify_ssa_operands (struct function *, gimple *stmt);
> extern void free_stmt_operands (struct function *, gimple *);
> -extern void update_stmt_operands (struct function *, gimple *);
> +extern void update_stmt_operands (struct function *, gimple *, bool);
> extern void swap_ssa_operands (gimple *, tree *, tree *);
> extern bool verify_imm_links (FILE *f, tree var);
>
> Index: gcc/tree-ssa-operands.c
> ===================================================================
> --- gcc/tree-ssa-operands.c 2018-08-28 11:25:46.242879876 +0100
> +++ gcc/tree-ssa-operands.c 2018-08-28 12:05:19.262917177 +0100
> @@ -99,6 +99,14 @@ #define opf_not_non_addressable (1 << 4)
> /* Operand is having its address taken. */
> #define opf_address_taken (1 << 5)
>
> +/* Operand must have no effect on code generation. This is used for
> + debug statements, and also for statements that a pass has no intention
> + of adding to the block in their current form. */
> +#define opf_transparent (1 << 6)
> +
> +/* Flags that must never be dropped. */
> +#define opf_sticky (opf_no_vops | opf_transparent)
> +
> /* Array for building all the use operands. */
> static vec<tree *> build_uses;
>
> @@ -590,7 +598,7 @@ get_mem_ref_operands (struct function *f
> /* If requested, add a USE operand for the base pointer. */
> get_expr_operands (fn, stmt, pptr,
> opf_non_addressable | opf_use
> - | (flags & (opf_no_vops|opf_not_non_addressable)));
> + | (flags & (opf_sticky | opf_not_non_addressable)));
> }
>
>
> @@ -605,11 +613,11 @@ get_tmr_operands (struct function *fn, g
>
> /* First record the real operands. */
> get_expr_operands (fn, stmt,
> - &TMR_BASE (expr), opf_use | (flags & opf_no_vops));
> + &TMR_BASE (expr), opf_use | (flags & opf_sticky));
> get_expr_operands (fn, stmt,
> - &TMR_INDEX (expr), opf_use | (flags & opf_no_vops));
> + &TMR_INDEX (expr), opf_use | (flags & opf_sticky));
> get_expr_operands (fn, stmt,
> - &TMR_INDEX2 (expr), opf_use | (flags & opf_no_vops));
> + &TMR_INDEX2 (expr), opf_use | (flags & opf_sticky));
>
> add_virtual_operand (fn, stmt, flags);
> }
> @@ -703,14 +711,14 @@ get_expr_operands (struct function *fn,
> enum tree_code code;
> enum tree_code_class codeclass;
> tree expr = *expr_p;
> - int uflags = opf_use;
> + int uflags = opf_use | (flags & opf_sticky);
> + gcc_checking_assert (!is_gimple_debug (stmt)
> + || ((flags & opf_no_vops)
> + && (flags & opf_transparent)));
>
> if (expr == NULL)
> return;
>
> - if (is_gimple_debug (stmt))
> - uflags |= (flags & opf_no_vops);
> -
> code = TREE_CODE (expr);
> codeclass = TREE_CODE_CLASS (code);
>
> @@ -723,7 +731,7 @@ get_expr_operands (struct function *fn,
> resolution). */
> if ((!(flags & opf_non_addressable)
> || (flags & opf_not_non_addressable))
> - && !is_gimple_debug (stmt))
> + && !(flags & opf_transparent))
> mark_address_taken (TREE_OPERAND (expr, 0));
>
> /* Otherwise, there may be variables referenced inside but there
> @@ -885,43 +893,50 @@ get_expr_operands (struct function *fn,
>
>
> /* Parse STMT looking for operands. When finished, the various
> - build_* operand vectors will have potential operands in them. */
> + build_* operand vectors will have potential operands in them.
> + TRANSPARENT_P as for update_stmt_operands. */
>
> static void
> -parse_ssa_operands (struct function *fn, gimple *stmt)
> +parse_ssa_operands (struct function *fn, gimple *stmt, bool transparent_p)
> {
> enum gimple_code code = gimple_code (stmt);
> size_t i, n, start = 0;
> + int flags = (transparent_p || code == GIMPLE_DEBUG
> + ? opf_no_vops | opf_transparent : 0);
>
> switch (code)
> {
> case GIMPLE_ASM:
> + /* Not supported yet (but could be if needed). */
> + gcc_assert (!transparent_p);
> get_asm_stmt_operands (fn, as_a <gasm *> (stmt));
> break;
>
> case GIMPLE_TRANSACTION:
> /* The start of a transaction is a memory barrier. */
> - add_virtual_operand (fn, stmt, opf_def | opf_use);
> + add_virtual_operand (fn, stmt, opf_def | opf_use | flags);
> break;
>
> case GIMPLE_DEBUG:
> if (gimple_debug_bind_p (stmt)
> && gimple_debug_bind_has_value_p (stmt))
> get_expr_operands (fn, stmt, gimple_debug_bind_get_value_ptr (stmt),
> - opf_use | opf_no_vops);
> + opf_use | flags);
> break;
>
> case GIMPLE_RETURN:
> - append_vuse (gimple_vop (fn));
> + if (!(flags & opf_no_vops))
> + append_vuse (gimple_vop (fn));
> goto do_default;
>
> case GIMPLE_CALL:
> /* Add call-clobbered operands, if needed. */
> - maybe_add_call_vops (fn, as_a <gcall *> (stmt));
> + if (!(flags & opf_no_vops))
> + maybe_add_call_vops (fn, as_a <gcall *> (stmt));
> /* FALLTHRU */
>
> case GIMPLE_ASSIGN:
> - get_expr_operands (fn, stmt, gimple_op_ptr (stmt, 0), opf_def);
> + get_expr_operands (fn, stmt, gimple_op_ptr (stmt, 0), opf_def | flags);
> start = 1;
> /* FALLTHRU */
>
> @@ -929,22 +944,23 @@ parse_ssa_operands (struct function *fn,
> do_default:
> n = gimple_num_ops (stmt);
> for (i = start; i < n; i++)
> - get_expr_operands (fn, stmt, gimple_op_ptr (stmt, i), opf_use);
> + get_expr_operands (fn, stmt, gimple_op_ptr (stmt, i), opf_use | flags);
> break;
> }
> }
>
>
> -/* Create an operands cache for STMT. */
> +/* Create an operands cache for STMT. TRANSPARENT_P as for
> + update_stmt_operands. */
>
> static void
> -build_ssa_operands (struct function *fn, gimple *stmt)
> +build_ssa_operands (struct function *fn, gimple *stmt, bool transparent_p)
> {
> /* Initially assume that the statement has no volatile operands. */
> gimple_set_has_volatile_ops (stmt, false);
>
> start_ssa_stmt_operands ();
> - parse_ssa_operands (fn, stmt);
> + parse_ssa_operands (fn, stmt, transparent_p);
> finalize_ssa_stmt_operands (fn, stmt);
> }
>
> @@ -963,7 +979,7 @@ verify_ssa_operands (struct function *fn
> /* build_ssa_operands w/o finalizing them. */
> gimple_set_has_volatile_ops (stmt, false);
> start_ssa_stmt_operands ();
> - parse_ssa_operands (fn, stmt);
> + parse_ssa_operands (fn, stmt, false);
>
> /* Now verify the built operands are the same as present in STMT. */
> def = gimple_vdef (stmt);
> @@ -1065,10 +1081,11 @@ free_stmt_operands (struct function *fn,
> }
>
>
> -/* Get the operands of statement STMT. */
> +/* Get the operands of statement STMT. TRANSPARENT_P says that opf_transparent
> + semantics should be used whatever form STMT happens to have. */
>
> void
> -update_stmt_operands (struct function *fn, gimple *stmt)
> +update_stmt_operands (struct function *fn, gimple *stmt, bool transparent_p)
> {
> /* If update_stmt_operands is called before SSA is initialized, do
> nothing. */
> @@ -1078,7 +1095,7 @@ update_stmt_operands (struct function *f
> timevar_push (TV_TREE_OPS);
>
> gcc_assert (gimple_modified_p (stmt));
> - build_ssa_operands (fn, stmt);
> + build_ssa_operands (fn, stmt, transparent_p);
> gimple_set_modified (stmt, false);
>
> timevar_pop (TV_TREE_OPS);
> Index: gcc/gimple-ssa.h
> ===================================================================
> --- gcc/gimple-ssa.h 2018-05-02 08:37:33.501751141 +0100
> +++ gcc/gimple-ssa.h 2018-08-28 12:05:19.262917177 +0100
> @@ -164,36 +164,42 @@ gimple_vdef_op (gimple *g)
> return NULL_DEF_OPERAND_P;
> }
>
> -/* Mark statement S as modified, and update it. */
> +/* Mark statement S as modified, and update it. TRANSPARENT_P is true
> + if the update must have no effect on code generation, in much the
> + same way as for debug statements. This means in particular that the
> + statement should not cause things to be marked addressable and should
> + not use virtual operands. */
>
> static inline void
> -update_stmt (gimple *s)
> +update_stmt (gimple *s, bool transparent_p = false)
> {
> if (gimple_has_ops (s))
> {
> gimple_set_modified (s, true);
> - update_stmt_operands (cfun, s);
> + update_stmt_operands (cfun, s, transparent_p);
> }
> }
>
> -/* Update statement S if it has been optimized. */
> +/* Update statement S if it has been optimized. TRANSPARENT_P is as for
> + update_stmt. */
>
> static inline void
> -update_stmt_if_modified (gimple *s)
> +update_stmt_if_modified (gimple *s, bool transparent_p = false)
> {
> if (gimple_modified_p (s))
> - update_stmt_operands (cfun, s);
> + update_stmt_operands (cfun, s, transparent_p);
> }
>
> -/* Mark statement S as modified, and update it. */
> +/* Mark statement S as modified, and update it. TRANSPARENT_P is as for
> + update_stmt. */
>
> static inline void
> -update_stmt_fn (struct function *fn, gimple *s)
> +update_stmt_fn (struct function *fn, gimple *s, bool transparent_p = false)
> {
> if (gimple_has_ops (s))
> {
> gimple_set_modified (s, true);
> - update_stmt_operands (fn, s);
> + update_stmt_operands (fn, s, transparent_p);
> }
> }
>
> Index: gcc/tree-vect-slp.c
> ===================================================================
> --- gcc/tree-vect-slp.c 2018-08-28 12:05:16.522940287 +0100
> +++ gcc/tree-vect-slp.c 2018-08-28 12:05:19.262917177 +0100
> @@ -2302,50 +2302,6 @@ vect_detect_hybrid_slp_stmts (slp_tree n
> vect_detect_hybrid_slp_stmts (child, i, stype);
> }
>
> -/* Helpers for vect_detect_hybrid_slp walking pattern stmt uses. */
> -
> -static tree
> -vect_detect_hybrid_slp_1 (tree *tp, int *, void *data)
> -{
> - walk_stmt_info *wi = (walk_stmt_info *)data;
> - loop_vec_info loop_vinfo = (loop_vec_info) wi->info;
> -
> - if (wi->is_lhs)
> - return NULL_TREE;
> -
> - stmt_vec_info def_stmt_info = loop_vinfo->lookup_def (*tp);
> - if (def_stmt_info && PURE_SLP_STMT (def_stmt_info))
> - {
> - if (dump_enabled_p ())
> - {
> - dump_printf_loc (MSG_NOTE, vect_location, "marking hybrid: ");
> - dump_gimple_stmt (MSG_NOTE, TDF_SLIM, def_stmt_info->stmt, 0);
> - }
> - STMT_SLP_TYPE (def_stmt_info) = hybrid;
> - }
> -
> - return NULL_TREE;
> -}
> -
> -static tree
> -vect_detect_hybrid_slp_2 (gimple_stmt_iterator *gsi, bool *handled,
> - walk_stmt_info *wi)
> -{
> - loop_vec_info loop_vinfo = (loop_vec_info) wi->info;
> - stmt_vec_info use_vinfo = loop_vinfo->lookup_stmt (gsi_stmt (*gsi));
> - /* If the stmt is in a SLP instance then this isn't a reason
> - to mark use definitions in other SLP instances as hybrid. */
> - if (! STMT_SLP_TYPE (use_vinfo)
> - && (STMT_VINFO_RELEVANT (use_vinfo)
> - || VECTORIZABLE_CYCLE_DEF (STMT_VINFO_DEF_TYPE (use_vinfo)))
> - && ! (gimple_code (gsi_stmt (*gsi)) == GIMPLE_PHI
> - && STMT_VINFO_DEF_TYPE (use_vinfo) == vect_reduction_def))
> - ;
> - else
> - *handled = true;
> - return NULL_TREE;
> -}
> -
> /* Find stmts that must be both vectorized and SLPed. */
>
> void
> @@ -2357,22 +2313,7 @@ vect_detect_hybrid_slp (loop_vec_info lo
>
> DUMP_VECT_SCOPE ("vect_detect_hybrid_slp");
>
> - /* First walk all pattern stmt in the loop and mark defs of uses as
> - hybrid because immediate uses in them are not recorded. */
> - vec_basic_block *vec_bb;
> - FOR_EACH_VEC_ELT (loop_vinfo->blocks, i, vec_bb)
> - FOR_EACH_VEC_BB_STMT (vec_bb, stmt_info)
> - if (is_pattern_stmt_p (stmt_info))
> - {
> - walk_stmt_info wi;
> - memset (&wi, 0, sizeof (wi));
> - wi.info = loop_vinfo;
> - gimple_stmt_iterator gsi = gsi_for_stmt (stmt_info->stmt);
> - walk_gimple_stmt (&gsi, vect_detect_hybrid_slp_2,
> - vect_detect_hybrid_slp_1, &wi);
> - }
> -
> - /* Then walk the SLP instance trees marking stmts with uses in
> + /* Walk the SLP instance trees marking stmts with uses in
> non-SLP stmts as hybrid, also propagating hybrid down the
> SLP tree, collecting the above info on-the-fly. */
> FOR_EACH_VEC_ELT (slp_instances, i, instance)
> Index: gcc/tree-vect-stmts.c
> ===================================================================
> --- gcc/tree-vect-stmts.c 2018-08-28 12:05:16.522940287 +0100
> +++ gcc/tree-vect-stmts.c 2018-08-28 12:05:19.266917143 +0100
> @@ -703,54 +703,13 @@ vect_mark_stmts_to_be_vectorized (loop_v
> break;
> }
>
> - if (is_pattern_stmt_p (stmt_vinfo))
> - {
> - /* Pattern statements are not inserted into the code, so
> - FOR_EACH_PHI_OR_STMT_USE optimizes their operands out, and we
> - have to scan the RHS or function arguments instead. */
> - if (gassign *assign = dyn_cast <gassign *> (stmt_vinfo->stmt))
> - {
> - enum tree_code rhs_code = gimple_assign_rhs_code (assign);
> - tree op = gimple_assign_rhs1 (assign);
> -
> - i = 1;
> - if (rhs_code == COND_EXPR && COMPARISON_CLASS_P (op))
> - {
> - if (!process_use (stmt_vinfo, TREE_OPERAND (op, 0),
> - loop_vinfo, relevant, &worklist, false)
> - || !process_use (stmt_vinfo, TREE_OPERAND (op, 1),
> - loop_vinfo, relevant, &worklist, false))
> - return false;
> - i = 2;
> - }
> - for (; i < gimple_num_ops (assign); i++)
> - {
> - op = gimple_op (assign, i);
> - if (TREE_CODE (op) == SSA_NAME
> - && !process_use (stmt_vinfo, op, loop_vinfo, relevant,
> - &worklist, false))
> - return false;
> - }
> - }
> - else if (gcall *call = dyn_cast <gcall *> (stmt_vinfo->stmt))
> - {
> - for (i = 0; i < gimple_call_num_args (call); i++)
> - {
> - tree arg = gimple_call_arg (call, i);
> - if (!process_use (stmt_vinfo, arg, loop_vinfo, relevant,
> - &worklist, false))
> - return false;
> - }
> - }
> - }
> - else
> - FOR_EACH_PHI_OR_STMT_USE (use_p, stmt_vinfo->stmt, iter, SSA_OP_USE)
> - {
> - tree op = USE_FROM_PTR (use_p);
> - if (!process_use (stmt_vinfo, op, loop_vinfo, relevant,
> - &worklist, false))
> - return false;
> - }
> + FOR_EACH_PHI_OR_STMT_USE (use_p, stmt_vinfo->stmt, iter, SSA_OP_USE)
> + {
> + tree op = USE_FROM_PTR (use_p);
> + if (!process_use (stmt_vinfo, op, loop_vinfo, relevant,
> + &worklist, false))
> + return false;
> + }
>
> if (STMT_VINFO_GATHER_SCATTER_P (stmt_vinfo))
> {
> @@ -10849,7 +10808,9 @@ vect_remove_dead_scalar_stmts (vec_info
> stmt_info = prev_stmt_info)
> {
> prev_stmt_info = stmt_info->prev;
> - if (!is_pattern_stmt_p (stmt_info))
> + if (is_pattern_stmt_p (stmt_info))
> + vinfo->remove_stmt (stmt_info);
> + else
> vect_maybe_remove_scalar_stmt (stmt_info);
> }
> }
> Index: gcc/tree-vectorizer.h
> ===================================================================
> --- gcc/tree-vectorizer.h 2018-08-28 12:05:16.522940287 +0100
> +++ gcc/tree-vectorizer.h 2018-08-28 12:05:19.266917143 +0100
> @@ -193,6 +193,8 @@ #define SLP_TREE_DEF_TYPE(S) (S)->def
> stmt_vec_info last () const { return m_last; }
>
> private:
> + void add_pattern_stmt_to_block (stmt_vec_info);
> +
> /* The block itself. */
> basic_block m_bb;
>
> Index: gcc/tree-vectorizer.c
> ===================================================================
> --- gcc/tree-vectorizer.c 2018-08-28 12:05:14.014961439 +0100
> +++ gcc/tree-vectorizer.c 2018-08-28 12:05:19.266917143 +0100
> @@ -459,6 +459,9 @@ vec_basic_block::add_to_end (stmt_vec_in
> stmt_info->block = this;
> stmt_info->prev = m_last;
> m_last = stmt_info;
> +
> + if (is_pattern_stmt_p (stmt_info))
> + add_pattern_stmt_to_block (stmt_info);
> }
>
> /* Add STMT_INFO to the block, inserting it before NEXT_STMT_INFO. */
> @@ -479,6 +482,34 @@ vec_basic_block::add_before (stmt_vec_in
> stmt_info->prev = next_stmt_info->prev;
> stmt_info->next = next_stmt_info;
> next_stmt_info->prev = stmt_info;
> +
> + if (is_pattern_stmt_p (stmt_info))
> + add_pattern_stmt_to_block (stmt_info);
> +}
> +
> +/* Record that pattern statement STMT_INFO has just been added to the
> + vec_basic_block. Adding it to the underlying basic_block would be
> + problematic because we need to be able to duplicate the original
> + scalar code in the middle of vectorization. Instead we just set the
> + statement's gimple_bb and link its SSA uses. */
> +
> +void
> +vec_basic_block::add_pattern_stmt_to_block (stmt_vec_info stmt_info)
> +{
> + gimple_set_bb (stmt_info->stmt, m_bb);
> + update_stmt (stmt_info->stmt, true);
> + gcc_assert (!gimple_vdef (stmt_info->stmt));
> + gcc_assert (!gimple_vuse (stmt_info->stmt));
> +}
> +
> +/* Record that STMT_INFO has been removed from its vec_basic_block.
> + Undo the effect of add_pattern_stmt_to_block. */
> +
> +static void
> +remove_pattern_stmt_from_block (stmt_vec_info stmt_info)
> +{
> + gimple_set_bb (stmt_info->stmt, NULL);
> + delink_stmt_imm_use (stmt_info->stmt);
> }
>
> /* Remove STMT_INFO from the block. */
> @@ -497,6 +528,9 @@ vec_basic_block::remove (stmt_vec_info s
> m_last = stmt_info->prev;
> stmt_info->block = NULL;
> stmt_info->prev = stmt_info->next = NULL;
> +
> + if (is_pattern_stmt_p (stmt_info))
> + remove_pattern_stmt_from_block (stmt_info);
> }
>
> /* Initialize the vec_info with kind KIND_IN and target cost data
> @@ -610,12 +644,37 @@ vec_info::lookup_def (tree name)
> stmt_vec_info
> vec_info::lookup_single_use (stmt_vec_info stmt_info)
> {
> - tree lhs = gimple_get_lhs (stmt_info->stmt);
> - use_operand_p dummy;
> - gimple *use_stmt;
> - if (single_imm_use (lhs, &dummy, &use_stmt))
> - return lookup_stmt (use_stmt);
> - return NULL;
> + gimple *stmts[2] = { stmt_info->stmt, NULL };
> + unsigned int num_stmts = 1;
> + /* If STMT_INFO replaces one of the original scalar statements,
> + check for uses of that statement's results too, to simulate the
> + effect of vect_stmt_to_be_vectorized. */
> + if (is_main_pattern_stmt_p (stmt_info))
> + stmts[num_stmts++] = STMT_VINFO_RELATED_STMT (stmt_info)->stmt;
> + stmt_vec_info result = NULL;
> + for (unsigned int i = 0; i < num_stmts; ++i)
> + {
> + use_operand_p use_p;
> + imm_use_iterator imm_iter;
> + tree lhs = gimple_get_lhs (stmts[i]);
> + FOR_EACH_IMM_USE_FAST (use_p, imm_iter, lhs)
> + {
> + gimple *use_stmt = USE_STMT (use_p);
> + if (is_gimple_debug (use_stmt))
> + continue;
> + stmt_vec_info use_stmt_info = lookup_stmt (use_stmt);
> + if (!use_stmt_info)
> + return NULL;
> + /* Ignore statements that have been replaced by a pattern
> + and consider only the replacement statement. */
> + if (STMT_VINFO_IN_PATTERN_P (use_stmt_info))
> + continue;
> + if (result)
> + return NULL;
> + result = use_stmt_info;
> + }
> + }
> + return result;
> }
>
> /* Return vectorization information about DR. */
> @@ -650,16 +709,18 @@ vec_info::move_dr (stmt_vec_info new_stm
> void
> vec_info::remove_stmt (stmt_vec_info stmt_info)
> {
> - gcc_assert (!stmt_info->pattern_stmt_p);
> set_vinfo_for_stmt (stmt_info->stmt, NULL);
> gimple_stmt_iterator si = gsi_for_stmt (stmt_info->stmt);
> unlink_stmt_vdef (stmt_info->stmt);
> - if (is_a <gphi *> (stmt_info->stmt))
> - remove_phi_node (&si, true);
> - else
> + if (!is_pattern_stmt_p (stmt_info))
> {
> - gsi_remove (&si, true);
> - release_defs (stmt_info->stmt);
> + if (is_a <gphi *> (stmt_info->stmt))
> + remove_phi_node (&si, true);
> + else
> + {
> + gsi_remove (&si, true);
> + release_defs (stmt_info->stmt);
> + }
> }
> stmt_info->block->remove (stmt_info);
> free_stmt_vec_info (stmt_info);
> @@ -749,12 +810,13 @@ vec_info::free_stmt_vec_infos (void)
> void
> vec_info::free_stmt_vec_info (stmt_vec_info stmt_info)
> {
> - if (stmt_info->pattern_stmt_p)
> + if (is_pattern_stmt_p (stmt_info))
> {
> - gimple_set_bb (stmt_info->stmt, NULL);
> tree lhs = gimple_get_lhs (stmt_info->stmt);
> if (lhs && TREE_CODE (lhs) == SSA_NAME)
> release_ssa_name (lhs);
> + if (stmt_info->block)
> + remove_pattern_stmt_from_block (stmt_info);
> }
>
> STMT_VINFO_SAME_ALIGN_REFS (stmt_info).release ();
> Index: gcc/tree-vect-patterns.c
> ===================================================================
> --- gcc/tree-vect-patterns.c 2018-08-28 12:05:16.518940320 +0100
> +++ gcc/tree-vect-patterns.c 2018-08-28 12:05:19.262917177 +0100
> @@ -106,7 +106,6 @@ vect_init_pattern_stmt (gimple *pattern_
> stmt_vec_info pattern_stmt_info = vinfo->lookup_stmt (pattern_stmt);
> if (pattern_stmt_info == NULL)
> pattern_stmt_info = orig_stmt_info->vinfo->add_stmt (pattern_stmt);
> - gimple_set_bb (pattern_stmt, gimple_bb (orig_stmt_info->stmt));
>
> pattern_stmt_info->pattern_stmt_p = true;
> STMT_VINFO_RELATED_STMT (pattern_stmt_info) = orig_stmt_info;
> @@ -412,11 +411,9 @@ vect_look_through_possible_promotion (ve
> break;
> caster = def_stmt_info;
>
> - /* Ignore pattern statements, since we don't link uses for them. */
> if (caster
> && single_use_p
> - && !STMT_VINFO_RELATED_STMT (caster)
> - && !has_single_use (res))
> + && !vinfo->lookup_single_use (caster))
> *single_use_p = false;
>
> gassign *assign = dyn_cast <gassign *> (def_stmt);