Support for aliasing with variable strides
Richard Sandiford
richard.sandiford@linaro.org
Fri Nov 17 22:19:00 GMT 2017
This patch adds runtime alias checks for loops with variable strides,
so that we can vectorise them even without a restrict qualifier.
There are several parts to doing this:
1) For accesses like:
x[i * n] += 1;
we need to check whether n (and thus the DR_STEP) is nonzero.
vect_analyze_data_ref_dependence records values that need to be
checked in this way, then prune_runtime_alias_test_list records a
bounds check on DR_STEP being outside the range [0, 0].
2) For accesses like:
x[i * n] = x[i * n + 1] + 1;
we simply need to test whether abs (n) >= 2.
prune_runtime_alias_test_list looks for cases like this and tries
to guess whether it is better to use this kind of check or a check
for non-overlapping ranges. (We could do an OR of the two conditions
at runtime, but that isn't implemented yet.)
3) Checks for overlapping ranges need to cope with variable strides.
At present the "length" of each segment in a range check is
represented as an offset from the base that lies outside the
touched range, in the same direction as DR_STEP. The length
can therefore be negative and is sometimes conservative.
With variable steps it's easier to reaon about if we split
this into two:
seg_len:
distance travelled from the first iteration of interest
to the last, e.g. DR_STEP * (VF - 1)
access_size:
the number of bytes accessed in each iteration
with access_size always being a positive constant and seg_len
possibly being variable. We can then combine alias checks
for two accesses that are a constant number of bytes apart by
adjusting the access size to account for the gap. This leaves
the segment length unchanged, which allows the check to be combined
with further accesses.
When seg_len is positive, the runtime alias check has the form:
base_a >= base_b + seg_len_b + access_size_b
|| base_b >= base_a + seg_len_a + access_size_a
In many accesses the base will be aligned to the access size, which
allows us to skip the addition:
base_a > base_b + seg_len_b
|| base_b > base_a + seg_len_a
A similar saving is possible with "negative" lengths.
The patch therefore tracks the alignment in addition to seg_len
and access_size.
Tested on aarch64-linux-gnu (with and without SVE), x86_64-linux-gnu
and powerpc64le-linux-gnu. OK to install?
Richard
PS. This is the last SVE patch for GCC 8!
2017-11-17 Richard Sandiford <richard.sandiford@linaro.org>
Alan Hayward <alan.hayward@arm.com>
David Sherwood <david.sherwood@arm.com>
gcc/
* tree-vectorizer.h (vec_lower_bound): New structure.
(_loop_vec_info): Add check_nonzero and lower_bounds.
(LOOP_VINFO_CHECK_NONZERO): New macro.
(LOOP_VINFO_LOWER_BOUNDS): Likewise.
(LOOP_REQUIRES_VERSIONING_FOR_ALIAS): Check lower_bounds too.
* tree-data-ref.h (dr_with_seg_len): Add access_size and align
fields. Make seg_len the distance travelled, not including the
access size.
(dr_direction_indicator): Declare.
(dr_zero_step_indicator): Likewise.
(dr_known_forward_stride_p): Likewise.
* tree-data-ref.c: Include stringpool.h, tree-vrp.h and
tree-ssanames.h.
(runtime_alias_check_p): Allow runtime alias checks with
variable strides.
(operator ==): Compare access_size and align.
(prune_runtime_alias_test_list): Rework for new distinction between
the access_size and seg_len.
(create_intersect_range_checks_index): Likewise. Cope with polynomial
segment lengths.
(get_segment_min_max): New function.
(create_intersect_range_checks): Use it.
(dr_step_indicator): New function.
(dr_direction_indicator): Likewise.
(dr_zero_step_indicator): Likewise.
(dr_known_forward_stride_p): Likewise.
* tree-loop-distribution.c (data_ref_segment_size): Return
DR_STEP * (niters - 1).
(compute_alias_check_pairs): Update call to the dr_with_seg_len
constructor.
* tree-vect-data-refs.c (vect_check_nonzero_value): New function.
(vect_preserves_scalar_order_p): New function, split out from...
(vect_analyze_data_ref_dependence): ...here. Check for zero steps.
(vect_vfa_segment_size): Return DR_STEP * (length_factor - 1).
(vect_vfa_access_size): New function.
(vect_vfa_align): Likewise.
(vect_compile_time_alias): Take access_size_a and access_b arguments.
(dump_lower_bound): New function.
(vect_check_lower_bound): Likewise.
(vect_small_gap_p): Likewise.
(vectorizable_with_step_bound_p): Likewise.
(vect_prune_runtime_alias_test_list): Ignore cross-iteration
depencies if the vectorization factor is 1. Convert the checks
for nonzero steps into checks on the bounds of DR_STEP. Try using
a bunds check for variable steps if the minimum required step is
relatively small. Update calls to the dr_with_seg_len
constructor and to vect_compile_time_alias.
* tree-vect-loop-manip.c (vect_create_cond_for_lower_bounds): New
function.
(vect_loop_versioning): Call it.
* tree-vect-loop.c (vect_analyze_loop_2): Clear LOOP_VINFO_LOWER_BOUNDS
when retrying.
(vect_estimate_min_profitable_iters): Account for any bounds checks.
gcc/testsuite/
* gcc.dg/vect/bb-slp-cond-1.c: Expect loop vectorization rather
than SLP vectorization.
* gcc.dg/vect/vect-alias-check-10.c: New test.
* gcc.dg/vect/vect-alias-check-11.c: Likewise.
* gcc.dg/vect/vect-alias-check-12.c: Likewise.
* gcc.dg/vect/vect-alias-check-8.c: Likewise.
* gcc.dg/vect/vect-alias-check-9.c: Likewise.
* gcc.target/aarch64/sve_strided_load_8.c: Likewise.
* gcc.target/aarch64/sve_var_stride_1.c: Likewise.
* gcc.target/aarch64/sve_var_stride_1.h: Likewise.
* gcc.target/aarch64/sve_var_stride_1_run.c: Likewise.
* gcc.target/aarch64/sve_var_stride_2.c: Likewise.
* gcc.target/aarch64/sve_var_stride_2_run.c: Likewise.
* gcc.target/aarch64/sve_var_stride_3.c: Likewise.
* gcc.target/aarch64/sve_var_stride_3_run.c: Likewise.
* gcc.target/aarch64/sve_var_stride_4.c: Likewise.
* gcc.target/aarch64/sve_var_stride_4_run.c: Likewise.
* gcc.target/aarch64/sve_var_stride_5.c: Likewise.
* gcc.target/aarch64/sve_var_stride_5_run.c: Likewise.
* gcc.target/aarch64/sve_var_stride_6.c: Likewise.
* gcc.target/aarch64/sve_var_stride_6_run.c: Likewise.
* gcc.target/aarch64/sve_var_stride_7.c: Likewise.
* gcc.target/aarch64/sve_var_stride_7_run.c: Likewise.
* gcc.target/aarch64/sve_var_stride_8.c: Likewise.
* gcc.target/aarch64/sve_var_stride_8_run.c: Likewise.
* gfortran.dg/vect/vect-alias-check-1.F90: Likewise.
Index: gcc/tree-vectorizer.h
===================================================================
*** gcc/tree-vectorizer.h 2017-11-17 22:07:58.227016438 +0000
--- gcc/tree-vectorizer.h 2017-11-17 22:11:53.273439843 +0000
*************** #define SLP_TREE_DEF_TYPE(S) (S)->def
*** 174,179 ****
--- 174,191 ----
loop to be valid. */
typedef std::pair<tree, tree> vec_object_pair;
+ /* Records that vectorization is only possible if abs (EXPR) >= MIN_VALUE.
+ UNSIGNED_P is true if we can assume that abs (EXPR) == EXPR. */
+ struct vec_lower_bound {
+ vec_lower_bound () {}
+ vec_lower_bound (tree e, bool u, poly_uint64 m)
+ : expr (e), unsigned_p (u), min_value (m) {}
+
+ tree expr;
+ bool unsigned_p;
+ poly_uint64 min_value;
+ };
+
/* Vectorizer state common between loop and basic-block vectorization. */
struct vec_info {
enum vec_kind { bb, loop };
*************** typedef struct _loop_vec_info : public v
*** 406,411 ****
--- 418,431 ----
/* Check that the addresses of each pair of objects is unequal. */
auto_vec<vec_object_pair> check_unequal_addrs;
+ /* List of values that are required to be nonzero. This is used to check
+ whether things like "x[i * n] += 1;" are safe and eventually gets added
+ to the checks for lower bounds below. */
+ auto_vec<tree> check_nonzero;
+
+ /* List of values that need to be checked for a minimum value. */
+ auto_vec<vec_lower_bound> lower_bounds;
+
/* Statements in the loop that have data references that are candidates for a
runtime (loop versioning) misalignment check. */
auto_vec<gimple *> may_misalign_stmts;
*************** #define LOOP_VINFO_MAY_MISALIGN_STMTS(L)
*** 514,519 ****
--- 534,541 ----
#define LOOP_VINFO_MAY_ALIAS_DDRS(L) (L)->may_alias_ddrs
#define LOOP_VINFO_COMP_ALIAS_DDRS(L) (L)->comp_alias_ddrs
#define LOOP_VINFO_CHECK_UNEQUAL_ADDRS(L) (L)->check_unequal_addrs
+ #define LOOP_VINFO_CHECK_NONZERO(L) (L)->check_nonzero
+ #define LOOP_VINFO_LOWER_BOUNDS(L) (L)->lower_bounds
#define LOOP_VINFO_GROUPED_STORES(L) (L)->grouped_stores
#define LOOP_VINFO_SLP_INSTANCES(L) (L)->slp_instances
#define LOOP_VINFO_SLP_UNROLLING_FACTOR(L) (L)->slp_unrolling_factor
*************** #define LOOP_REQUIRES_VERSIONING_FOR_ALI
*** 534,540 ****
((L)->may_misalign_stmts.length () > 0)
#define LOOP_REQUIRES_VERSIONING_FOR_ALIAS(L) \
((L)->comp_alias_ddrs.length () > 0 \
! || (L)->check_unequal_addrs.length () > 0)
#define LOOP_REQUIRES_VERSIONING_FOR_NITERS(L) \
(LOOP_VINFO_NITERS_ASSUMPTIONS (L))
#define LOOP_REQUIRES_VERSIONING(L) \
--- 556,563 ----
((L)->may_misalign_stmts.length () > 0)
#define LOOP_REQUIRES_VERSIONING_FOR_ALIAS(L) \
((L)->comp_alias_ddrs.length () > 0 \
! || (L)->check_unequal_addrs.length () > 0 \
! || (L)->lower_bounds.length () > 0)
#define LOOP_REQUIRES_VERSIONING_FOR_NITERS(L) \
(LOOP_VINFO_NITERS_ASSUMPTIONS (L))
#define LOOP_REQUIRES_VERSIONING(L) \
Index: gcc/tree-data-ref.h
===================================================================
*** gcc/tree-data-ref.h 2017-11-17 21:44:42.168891072 +0000
--- gcc/tree-data-ref.h 2017-11-17 22:11:53.270650578 +0000
*************** typedef struct data_reference *data_refe
*** 203,213 ****
struct dr_with_seg_len
{
! dr_with_seg_len (data_reference_p d, tree len)
! : dr (d), seg_len (len) {}
data_reference_p dr;
tree seg_len;
};
/* This struct contains two dr_with_seg_len objects with aliasing data
--- 203,222 ----
struct dr_with_seg_len
{
! dr_with_seg_len (data_reference_p d, tree len, unsigned HOST_WIDE_INT size,
! unsigned int a)
! : dr (d), seg_len (len), access_size (size), align (a) {}
data_reference_p dr;
+ /* The offset of the last access that needs to be checked minus
+ the offset of the first. */
tree seg_len;
+ /* A value that, when added to abs (SEG_LEN), gives the total number of
+ bytes in the segment. */
+ poly_uint64 access_size;
+ /* The minimum common alignment of DR's start address, SEG_LEN and
+ ACCESS_SIZE. */
+ unsigned int align;
};
/* This struct contains two dr_with_seg_len objects with aliasing data
*************** extern void prune_runtime_alias_test_lis
*** 475,480 ****
--- 484,493 ----
poly_uint64);
extern void create_runtime_alias_checks (struct loop *,
vec<dr_with_seg_len_pair_t> *, tree*);
+ extern tree dr_direction_indicator (struct data_reference *);
+ extern tree dr_zero_step_indicator (struct data_reference *);
+ extern bool dr_known_forward_stride_p (struct data_reference *);
+
/* Return true when the base objects of data references A and B are
the same memory object. */
Index: gcc/tree-data-ref.c
===================================================================
*** gcc/tree-data-ref.c 2017-11-17 21:48:20.720851310 +0000
--- gcc/tree-data-ref.c 2017-11-17 22:11:53.270650578 +0000
*************** Software Foundation; either version 3, o
*** 95,100 ****
--- 95,103 ----
#include "tree-affine.h"
#include "params.h"
#include "builtins.h"
+ #include "stringpool.h"
+ #include "tree-vrp.h"
+ #include "tree-ssanames.h"
static struct datadep_stats
{
*************** runtime_alias_check_p (ddr_p ddr, struct
*** 1307,1324 ****
return false;
}
- /* FORNOW: We don't support creating runtime alias tests for non-constant
- step. */
- if (TREE_CODE (DR_STEP (DDR_A (ddr))) != INTEGER_CST
- || TREE_CODE (DR_STEP (DDR_B (ddr))) != INTEGER_CST)
- {
- if (dump_enabled_p ())
- dump_printf (MSG_MISSED_OPTIMIZATION,
- "runtime alias check not supported for non-constant "
- "step\n");
- return false;
- }
-
return true;
}
--- 1310,1315 ----
*************** runtime_alias_check_p (ddr_p ddr, struct
*** 1333,1343 ****
operator == (const dr_with_seg_len& d1,
const dr_with_seg_len& d2)
{
! return operand_equal_p (DR_BASE_ADDRESS (d1.dr),
! DR_BASE_ADDRESS (d2.dr), 0)
! && data_ref_compare_tree (DR_OFFSET (d1.dr), DR_OFFSET (d2.dr)) == 0
! && data_ref_compare_tree (DR_INIT (d1.dr), DR_INIT (d2.dr)) == 0
! && data_ref_compare_tree (d1.seg_len, d2.seg_len) == 0;
}
/* Comparison function for sorting objects of dr_with_seg_len_pair_t
--- 1324,1336 ----
operator == (const dr_with_seg_len& d1,
const dr_with_seg_len& d2)
{
! return (operand_equal_p (DR_BASE_ADDRESS (d1.dr),
! DR_BASE_ADDRESS (d2.dr), 0)
! && data_ref_compare_tree (DR_OFFSET (d1.dr), DR_OFFSET (d2.dr)) == 0
! && data_ref_compare_tree (DR_INIT (d1.dr), DR_INIT (d2.dr)) == 0
! && data_ref_compare_tree (d1.seg_len, d2.seg_len) == 0
! && must_eq (d1.access_size, d2.access_size)
! && d1.align == d2.align);
}
/* Comparison function for sorting objects of dr_with_seg_len_pair_t
*************** comp_dr_with_seg_len_pair (const void *p
*** 1417,1423 ****
void
prune_runtime_alias_test_list (vec<dr_with_seg_len_pair_t> *alias_pairs,
! poly_uint64 factor)
{
/* Sort the collected data ref pairs so that we can scan them once to
combine all possible aliasing checks. */
--- 1410,1416 ----
void
prune_runtime_alias_test_list (vec<dr_with_seg_len_pair_t> *alias_pairs,
! poly_uint64)
{
/* Sort the collected data ref pairs so that we can scan them once to
combine all possible aliasing checks. */
*************** prune_runtime_alias_test_list (vec<dr_wi
*** 1463,1468 ****
--- 1456,1463 ----
}
poly_int64 init_a1, init_a2;
+ /* Only consider cases in which the distance between the initial
+ DR_A1 and the initial DR_A2 is known at compile time. */
if (!operand_equal_p (DR_BASE_ADDRESS (dr_a1->dr),
DR_BASE_ADDRESS (dr_a2->dr), 0)
|| !operand_equal_p (DR_OFFSET (dr_a1->dr),
*************** prune_runtime_alias_test_list (vec<dr_wi
*** 1482,1622 ****
std::swap (init_a1, init_a2);
}
! /* Only merge const step data references. */
! poly_int64 step_a1, step_a2;
! if (!poly_int_tree_p (DR_STEP (dr_a1->dr), &step_a1)
! || !poly_int_tree_p (DR_STEP (dr_a2->dr), &step_a2))
! continue;
! bool neg_step = may_lt (step_a1, 0) || may_lt (step_a2, 0);
! /* DR_A1 and DR_A2 must go in the same direction. */
! if (neg_step && (may_gt (step_a1, 0) || may_gt (step_a2, 0)))
! continue;
! poly_uint64 seg_len_a1 = 0, seg_len_a2 = 0;
! bool const_seg_len_a1 = poly_int_tree_p (dr_a1->seg_len,
! &seg_len_a1);
! bool const_seg_len_a2 = poly_int_tree_p (dr_a2->seg_len,
! &seg_len_a2);
!
! /* We need to compute merged segment length at compilation time for
! dr_a1 and dr_a2, which is impossible if either one has non-const
! segment length. */
! if ((!const_seg_len_a1 || !const_seg_len_a2)
! && may_ne (step_a1, step_a2))
! continue;
! bool do_remove = false;
! poly_uint64 diff = init_a2 - init_a1;
! poly_uint64 min_seg_len_b;
! tree new_seg_len;
! if (!poly_int_tree_p (dr_b1->seg_len, &min_seg_len_b))
{
! tree step_b = DR_STEP (dr_b1->dr);
! if (!tree_fits_shwi_p (step_b))
continue;
- min_seg_len_b = factor * abs_hwi (tree_to_shwi (step_b));
- }
! /* Now we try to merge alias check dr_a1 & dr_b and dr_a2 & dr_b.
! Case A:
! check if the following condition is satisfied:
! DIFF - SEGMENT_LENGTH_A < SEGMENT_LENGTH_B
! where DIFF = DR_A2_INIT - DR_A1_INIT. However,
! SEGMENT_LENGTH_A or SEGMENT_LENGTH_B may not be constant so we
! have to make a best estimation. We can get the minimum value
! of SEGMENT_LENGTH_B as a constant, represented by MIN_SEG_LEN_B,
! then either of the following two conditions can guarantee the
! one above:
!
! 1: DIFF <= MIN_SEG_LEN_B
! 2: DIFF - SEGMENT_LENGTH_A < MIN_SEG_LEN_B
! Because DIFF - SEGMENT_LENGTH_A is done in sizetype, we need
! to take care of wrapping behavior in it.
!
! Case B:
! If the left segment does not extend beyond the start of the
! right segment the new segment length is that of the right
! plus the segment distance. The condition is like:
!
! DIFF >= SEGMENT_LENGTH_A ;SEGMENT_LENGTH_A is a constant.
!
! Note 1: Case A.2 and B combined together effectively merges every
! dr_a1 & dr_b and dr_a2 & dr_b when SEGMENT_LENGTH_A is const.
!
! Note 2: Above description is based on positive DR_STEP, we need to
! take care of negative DR_STEP for wrapping behavior. See PR80815
! for more information. */
! if (neg_step)
! {
! /* Adjust diff according to access size of both references. */
! diff += tree_to_poly_uint64
! (TYPE_SIZE_UNIT (TREE_TYPE (DR_REF (dr_a2->dr))));
! diff -= tree_to_poly_uint64
! (TYPE_SIZE_UNIT (TREE_TYPE (DR_REF (dr_a1->dr))));
! /* Case A.1. */
! if (must_le (diff, min_seg_len_b)
! /* Case A.2 and B combined. */
! || const_seg_len_a2)
! {
! if (const_seg_len_a1 || const_seg_len_a2)
! new_seg_len
! = build_int_cstu (sizetype,
! lower_bound (seg_len_a1 - diff,
! seg_len_a2));
! else
! new_seg_len
! = size_binop (MINUS_EXPR, dr_a2->seg_len,
! build_int_cstu (sizetype, diff));
! dr_a2->seg_len = new_seg_len;
! do_remove = true;
! }
}
- else
- {
- /* Case A.1. */
- if (must_le (diff, min_seg_len_b)
- /* Case A.2 and B combined. */
- || const_seg_len_a1)
- {
- if (const_seg_len_a1 && const_seg_len_a2)
- new_seg_len
- = build_int_cstu (sizetype,
- upper_bound (seg_len_a2 + diff,
- seg_len_a1));
- else
- new_seg_len
- = size_binop (PLUS_EXPR, dr_a2->seg_len,
- build_int_cstu (sizetype, diff));
! dr_a1->seg_len = new_seg_len;
! do_remove = true;
! }
! }
! if (do_remove)
{
! if (dump_enabled_p ())
! {
! dump_printf (MSG_NOTE, "merging ranges for ");
! dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_a1->dr));
! dump_printf (MSG_NOTE, ", ");
! dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_b1->dr));
! dump_printf (MSG_NOTE, " and ");
! dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_a2->dr));
! dump_printf (MSG_NOTE, ", ");
! dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_b2->dr));
! dump_printf (MSG_NOTE, "\n");
! }
! alias_pairs->ordered_remove (neg_step ? i - 1 : i);
! i--;
}
}
}
}
--- 1477,1555 ----
std::swap (init_a1, init_a2);
}
! /* Work out what the segment length would be if we did combine
! DR_A1 and DR_A2:
! - If DR_A1 and DR_A2 have equal lengths, that length is
! also the combined length.
! - If DR_A1 and DR_A2 both have negative "lengths", the combined
! length is the lower bound on those lengths.
! - If DR_A1 and DR_A2 both have positive lengths, the combined
! length is the upper bound on those lengths.
! Other cases are unlikely to give a useful combination.
! The lengths both have sizetype, so the sign is taken from
! the step instead. */
! if (!operand_equal_p (dr_a1->seg_len, dr_a2->seg_len, 0))
{
! poly_uint64 seg_len_a1, seg_len_a2;
! if (!poly_int_tree_p (dr_a1->seg_len, &seg_len_a1)
! || !poly_int_tree_p (dr_a2->seg_len, &seg_len_a2))
continue;
! tree indicator_a = dr_direction_indicator (dr_a1->dr);
! if (TREE_CODE (indicator_a) != INTEGER_CST)
! continue;
! tree indicator_b = dr_direction_indicator (dr_a2->dr);
! if (TREE_CODE (indicator_b) != INTEGER_CST)
! continue;
! int sign_a = tree_int_cst_sgn (indicator_a);
! int sign_b = tree_int_cst_sgn (indicator_b);
! poly_uint64 new_seg_len;
! if (sign_a <= 0 && sign_b <= 0)
! new_seg_len = lower_bound (seg_len_a1, seg_len_a2);
! else if (sign_a >= 0 && sign_b >= 0)
! new_seg_len = upper_bound (seg_len_a1, seg_len_a2);
! else
! continue;
! dr_a1->seg_len = build_int_cst (TREE_TYPE (dr_a1->seg_len),
! new_seg_len);
! dr_a1->align = MIN (dr_a1->align, known_alignment (new_seg_len));
}
! /* This is always positive due to the swap above. */
! poly_uint64 diff = init_a2 - init_a1;
! /* The new check will start at DR_A1. Make sure that its access
! size encompasses the initial DR_A2. */
! if (may_lt (dr_a1->access_size, diff + dr_a2->access_size))
{
! dr_a1->access_size = upper_bound (dr_a1->access_size,
! diff + dr_a2->access_size);
! unsigned int new_align = known_alignment (dr_a1->access_size);
! dr_a1->align = MIN (dr_a1->align, new_align);
}
+ if (dump_enabled_p ())
+ {
+ dump_printf (MSG_NOTE, "merging ranges for ");
+ dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_a1->dr));
+ dump_printf (MSG_NOTE, ", ");
+ dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_b1->dr));
+ dump_printf (MSG_NOTE, " and ");
+ dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_a2->dr));
+ dump_printf (MSG_NOTE, ", ");
+ dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_b2->dr));
+ dump_printf (MSG_NOTE, "\n");
+ }
+ alias_pairs->ordered_remove (i);
+ i--;
}
}
}
*************** create_intersect_range_checks_index (str
*** 1656,1662 ****
|| DR_NUM_DIMENSIONS (dr_a.dr) != DR_NUM_DIMENSIONS (dr_b.dr))
return false;
! if (!tree_fits_uhwi_p (dr_a.seg_len) || !tree_fits_uhwi_p (dr_b.seg_len))
return false;
if (!tree_fits_shwi_p (DR_STEP (dr_a.dr)))
--- 1589,1597 ----
|| DR_NUM_DIMENSIONS (dr_a.dr) != DR_NUM_DIMENSIONS (dr_b.dr))
return false;
! poly_uint64 seg_len1, seg_len2;
! if (!poly_int_tree_p (dr_a.seg_len, &seg_len1)
! || !poly_int_tree_p (dr_b.seg_len, &seg_len2))
return false;
if (!tree_fits_shwi_p (DR_STEP (dr_a.dr)))
*************** create_intersect_range_checks_index (str
*** 1671,1689 ****
gcc_assert (TREE_CODE (DR_STEP (dr_a.dr)) == INTEGER_CST);
bool neg_step = tree_int_cst_compare (DR_STEP (dr_a.dr), size_zero_node) < 0;
! unsigned HOST_WIDE_INT abs_step
! = absu_hwi (tree_to_shwi (DR_STEP (dr_a.dr)));
- unsigned HOST_WIDE_INT seg_len1 = tree_to_uhwi (dr_a.seg_len);
- unsigned HOST_WIDE_INT seg_len2 = tree_to_uhwi (dr_b.seg_len);
/* Infer the number of iterations with which the memory segment is accessed
by DR. In other words, alias is checked if memory segment accessed by
DR_A in some iterations intersect with memory segment accessed by DR_B
in the same amount iterations.
Note segnment length is a linear function of number of iterations with
DR_STEP as the coefficient. */
! unsigned HOST_WIDE_INT niter_len1 = (seg_len1 + abs_step - 1) / abs_step;
! unsigned HOST_WIDE_INT niter_len2 = (seg_len2 + abs_step - 1) / abs_step;
unsigned int i;
for (i = 0; i < DR_NUM_DIMENSIONS (dr_a.dr); i++)
--- 1606,1647 ----
gcc_assert (TREE_CODE (DR_STEP (dr_a.dr)) == INTEGER_CST);
bool neg_step = tree_int_cst_compare (DR_STEP (dr_a.dr), size_zero_node) < 0;
! unsigned HOST_WIDE_INT abs_step = tree_to_shwi (DR_STEP (dr_a.dr));
! if (neg_step)
! {
! abs_step = -abs_step;
! seg_len1 = -seg_len1;
! seg_len2 = -seg_len2;
! }
! else
! {
! /* Include the access size in the length, so that we only have one
! tree addition below. */
! seg_len1 += dr_a.access_size;
! seg_len2 += dr_b.access_size;
! }
/* Infer the number of iterations with which the memory segment is accessed
by DR. In other words, alias is checked if memory segment accessed by
DR_A in some iterations intersect with memory segment accessed by DR_B
in the same amount iterations.
Note segnment length is a linear function of number of iterations with
DR_STEP as the coefficient. */
! poly_uint64 niter_len1, niter_len2;
! if (!can_div_trunc_p (seg_len1 + abs_step - 1, abs_step, &niter_len1)
! || !can_div_trunc_p (seg_len2 + abs_step - 1, abs_step, &niter_len2))
! return false;
!
! poly_uint64 niter_access1 = 0, niter_access2 = 0;
! if (neg_step)
! {
! /* Divide each access size by the byte step, rounding up. */
! if (!can_div_trunc_p (dr_a.access_size - abs_step - 1,
! abs_step, &niter_access1)
! || !can_div_trunc_p (dr_b.access_size + abs_step - 1,
! abs_step, &niter_access2))
! return false;
! }
unsigned int i;
for (i = 0; i < DR_NUM_DIMENSIONS (dr_a.dr); i++)
*************** create_intersect_range_checks_index (str
*** 1734,1745 ****
/* Adjust ranges for negative step. */
if (neg_step)
{
! min1 = fold_build2 (MINUS_EXPR, TREE_TYPE (min1), max1, idx_step);
! max1 = fold_build2 (MINUS_EXPR, TREE_TYPE (min1),
! CHREC_LEFT (access1), idx_step);
! min2 = fold_build2 (MINUS_EXPR, TREE_TYPE (min2), max2, idx_step);
! max2 = fold_build2 (MINUS_EXPR, TREE_TYPE (min2),
! CHREC_LEFT (access2), idx_step);
}
tree part_cond_expr
= fold_build2 (TRUTH_OR_EXPR, boolean_type_node,
--- 1692,1713 ----
/* Adjust ranges for negative step. */
if (neg_step)
{
! /* IDX_LEN1 and IDX_LEN2 are negative in this case. */
! std::swap (min1, max1);
! std::swap (min2, max2);
!
! /* As with the lengths just calculated, we've measured the access
! sizes in iterations, so multiply them by the index step. */
! tree idx_access1
! = fold_build2 (MULT_EXPR, TREE_TYPE (min1), idx_step,
! build_int_cst (TREE_TYPE (min1), niter_access1));
! tree idx_access2
! = fold_build2 (MULT_EXPR, TREE_TYPE (min2), idx_step,
! build_int_cst (TREE_TYPE (min2), niter_access2));
!
! /* MINUS_EXPR because the above values are negative. */
! max1 = fold_build2 (MINUS_EXPR, TREE_TYPE (max1), max1, idx_access1);
! max2 = fold_build2 (MINUS_EXPR, TREE_TYPE (max2), max2, idx_access2);
}
tree part_cond_expr
= fold_build2 (TRUTH_OR_EXPR, boolean_type_node,
*************** create_intersect_range_checks_index (str
*** 1754,1759 ****
--- 1722,1810 ----
return true;
}
+ /* If ALIGN is nonzero, set up *SEQ_MIN_OUT and *SEQ_MAX_OUT so that for
+ every address ADDR accessed by D:
+
+ *SEQ_MIN_OUT <= ADDR (== ADDR & -ALIGN) <= *SEQ_MAX_OUT
+
+ In this case, every element accessed by D is aligned to at least
+ ALIGN bytes.
+
+ If ALIGN is zero then instead set *SEG_MAX_OUT so that:
+
+ *SEQ_MIN_OUT <= ADDR < *SEQ_MAX_OUT. */
+
+ static void
+ get_segment_min_max (const dr_with_seg_len &d, tree *seg_min_out,
+ tree *seg_max_out, HOST_WIDE_INT align)
+ {
+ /* Each access has the following pattern:
+
+ <- |seg_len| ->
+ <--- A: -ve step --->
+ +-----+-------+-----+-------+-----+
+ | n-1 | ,.... | 0 | ..... | n-1 |
+ +-----+-------+-----+-------+-----+
+ <--- B: +ve step --->
+ <- |seg_len| ->
+ |
+ base address
+
+ where "n" is the number of scalar iterations covered by the segment.
+ (This should be VF for a particular pair if we know that both steps
+ are the same, otherwise it will be the full number of scalar loop
+ iterations.)
+
+ A is the range of bytes accessed when the step is negative,
+ B is the range when the step is positive.
+
+ If the access size is "access_size" bytes, the lowest addressed byte is:
+
+ base + (step < 0 ? seg_len : 0) [LB]
+
+ and the highest addressed byte is always below:
+
+ base + (step < 0 ? 0 : seg_len) + access_size [UB]
+
+ Thus:
+
+ LB <= ADDR < UB
+
+ If ALIGN is nonzero, all three values are aligned to at least ALIGN
+ bytes, so:
+
+ LB <= ADDR <= UB - ALIGN
+
+ where "- ALIGN" folds naturally with the "+ access_size" and often
+ cancels it out.
+
+ We don't try to simplify LB and UB beyond this (e.g. by using
+ MIN and MAX based on whether seg_len rather than the stride is
+ negative) because it is possible for the absolute size of the
+ segment to overflow the range of a ssize_t.
+
+ Keeping the pointer_plus outside of the cond_expr should allow
+ the cond_exprs to be shared with other alias checks. */
+ tree indicator = dr_direction_indicator (d.dr);
+ tree neg_step = fold_build2 (LT_EXPR, boolean_type_node,
+ fold_convert (ssizetype, indicator),
+ ssize_int (0));
+ tree addr_base = fold_build_pointer_plus (DR_BASE_ADDRESS (d.dr),
+ DR_OFFSET (d.dr));
+ addr_base = fold_build_pointer_plus (addr_base, DR_INIT (d.dr));
+ tree seg_len = fold_convert (sizetype, d.seg_len);
+
+ tree min_reach = fold_build3 (COND_EXPR, sizetype, neg_step,
+ seg_len, size_zero_node);
+ tree max_reach = fold_build3 (COND_EXPR, sizetype, neg_step,
+ size_zero_node, seg_len);
+ max_reach = fold_build2 (PLUS_EXPR, sizetype, max_reach,
+ size_int (d.access_size - align));
+
+ *seg_min_out = fold_build_pointer_plus (addr_base, min_reach);
+ *seg_max_out = fold_build_pointer_plus (addr_base, max_reach);
+ }
+
/* Given two data references and segment lengths described by DR_A and DR_B,
create expression checking if the two addresses ranges intersect with
each other:
*************** create_intersect_range_checks (struct lo
*** 1770,1812 ****
if (create_intersect_range_checks_index (loop, cond_expr, dr_a, dr_b))
return;
! tree segment_length_a = dr_a.seg_len;
! tree segment_length_b = dr_b.seg_len;
! tree addr_base_a = DR_BASE_ADDRESS (dr_a.dr);
! tree addr_base_b = DR_BASE_ADDRESS (dr_b.dr);
! tree offset_a = DR_OFFSET (dr_a.dr), offset_b = DR_OFFSET (dr_b.dr);
!
! offset_a = fold_build2 (PLUS_EXPR, TREE_TYPE (offset_a),
! offset_a, DR_INIT (dr_a.dr));
! offset_b = fold_build2 (PLUS_EXPR, TREE_TYPE (offset_b),
! offset_b, DR_INIT (dr_b.dr));
! addr_base_a = fold_build_pointer_plus (addr_base_a, offset_a);
! addr_base_b = fold_build_pointer_plus (addr_base_b, offset_b);
!
! tree seg_a_min = addr_base_a;
! tree seg_a_max = fold_build_pointer_plus (addr_base_a, segment_length_a);
! /* For negative step, we need to adjust address range by TYPE_SIZE_UNIT
! bytes, e.g., int a[3] -> a[1] range is [a+4, a+16) instead of
! [a, a+12) */
! if (tree_int_cst_compare (DR_STEP (dr_a.dr), size_zero_node) < 0)
! {
! tree unit_size = TYPE_SIZE_UNIT (TREE_TYPE (DR_REF (dr_a.dr)));
! seg_a_min = fold_build_pointer_plus (seg_a_max, unit_size);
! seg_a_max = fold_build_pointer_plus (addr_base_a, unit_size);
! }
!
! tree seg_b_min = addr_base_b;
! tree seg_b_max = fold_build_pointer_plus (addr_base_b, segment_length_b);
! if (tree_int_cst_compare (DR_STEP (dr_b.dr), size_zero_node) < 0)
! {
! tree unit_size = TYPE_SIZE_UNIT (TREE_TYPE (DR_REF (dr_b.dr)));
! seg_b_min = fold_build_pointer_plus (seg_b_max, unit_size);
! seg_b_max = fold_build_pointer_plus (addr_base_b, unit_size);
}
*cond_expr
= fold_build2 (TRUTH_OR_EXPR, boolean_type_node,
! fold_build2 (LE_EXPR, boolean_type_node, seg_a_max, seg_b_min),
! fold_build2 (LE_EXPR, boolean_type_node, seg_b_max, seg_a_min));
}
/* Create a conditional expression that represents the run-time checks for
--- 1821,1868 ----
if (create_intersect_range_checks_index (loop, cond_expr, dr_a, dr_b))
return;
! unsigned HOST_WIDE_INT min_align;
! tree_code cmp_code;
! if (TREE_CODE (DR_STEP (dr_a.dr)) == INTEGER_CST
! && TREE_CODE (DR_STEP (dr_b.dr)) == INTEGER_CST)
! {
! /* In this case adding access_size to seg_len is likely to give
! a simple X * step, where X is either the number of scalar
! iterations or the vectorization factor. We're better off
! keeping that, rather than subtracting an alignment from it.
!
! In this case the maximum values are exclusive and so there is
! no alias if the maximum of one segment equals the minimum
! of another. */
! min_align = 0;
! cmp_code = LE_EXPR;
}
+ else
+ {
+ /* Calculate the minimum alignment shared by all four pointers,
+ then arrange for this alignment to be subtracted from the
+ exclusive maximum values to get inclusive maximum values.
+ This "- min_align" is cumulative with a "+ access_size"
+ in the calculation of the maximum values. In the best
+ (and common) case, the two cancel each other out, leaving
+ us with an inclusive bound based only on seg_len. In the
+ worst case we're simply adding a smaller number than before.
+
+ Because the maximum values are inclusive, there is an alias
+ if the maximum value of one segment is equal to the minimum
+ value of the other. */
+ min_align = MIN (dr_a.align, dr_b.align);
+ cmp_code = LT_EXPR;
+ }
+
+ tree seg_a_min, seg_a_max, seg_b_min, seg_b_max;
+ get_segment_min_max (dr_a, &seg_a_min, &seg_a_max, min_align);
+ get_segment_min_max (dr_b, &seg_b_min, &seg_b_max, min_align);
+
*cond_expr
= fold_build2 (TRUTH_OR_EXPR, boolean_type_node,
! fold_build2 (cmp_code, boolean_type_node, seg_a_max, seg_b_min),
! fold_build2 (cmp_code, boolean_type_node, seg_b_max, seg_a_min));
}
/* Create a conditional expression that represents the run-time checks for
*************** free_data_refs (vec<data_reference_p> da
*** 5277,5279 ****
--- 5333,5422 ----
free_data_ref (dr);
datarefs.release ();
}
+
+ /* Common routine implementing both dr_direction_indicator and
+ dr_zero_step_indicator. Return USEFUL_MIN if the indicator is known
+ to be >= USEFUL_MIN and -1 if the indicator is known to be negative.
+ Return the step as the indicator otherwise. */
+
+ static tree
+ dr_step_indicator (struct data_reference *dr, int useful_min)
+ {
+ tree step = DR_STEP (dr);
+ STRIP_NOPS (step);
+ /* Look for cases where the step is scaled by a positive constant
+ integer, which will often be the access size. If the multiplication
+ doesn't change the sign (due to overflow effects) then we can
+ test the unscaled value instead. */
+ if (TREE_CODE (step) == MULT_EXPR
+ && TREE_CODE (TREE_OPERAND (step, 1)) == INTEGER_CST
+ && tree_int_cst_sgn (TREE_OPERAND (step, 1)) > 0)
+ {
+ tree factor = TREE_OPERAND (step, 1);
+ step = TREE_OPERAND (step, 0);
+
+ /* Strip widening and truncating conversions as well as nops. */
+ if (CONVERT_EXPR_P (step)
+ && INTEGRAL_TYPE_P (TREE_TYPE (TREE_OPERAND (step, 0))))
+ step = TREE_OPERAND (step, 0);
+ tree type = TREE_TYPE (step);
+
+ /* Get the range of step values that would not cause overflow. */
+ widest_int minv = (wi::to_widest (TYPE_MIN_VALUE (ssizetype))
+ / wi::to_widest (factor));
+ widest_int maxv = (wi::to_widest (TYPE_MAX_VALUE (ssizetype))
+ / wi::to_widest (factor));
+
+ /* Get the range of values that the unconverted step actually has. */
+ wide_int step_min, step_max;
+ if (TREE_CODE (step) != SSA_NAME
+ || get_range_info (step, &step_min, &step_max) != VR_RANGE)
+ {
+ step_min = wi::to_wide (TYPE_MIN_VALUE (type));
+ step_max = wi::to_wide (TYPE_MAX_VALUE (type));
+ }
+
+ /* Check whether the unconverted step has an acceptable range. */
+ signop sgn = TYPE_SIGN (type);
+ if (wi::les_p (minv, widest_int::from (step_min, sgn))
+ && wi::ges_p (maxv, widest_int::from (step_max, sgn)))
+ {
+ if (wi::ge_p (step_min, useful_min, sgn))
+ return ssize_int (useful_min);
+ else if (wi::lt_p (step_max, 0, sgn))
+ return ssize_int (-1);
+ else
+ return fold_convert (ssizetype, step);
+ }
+ }
+ return DR_STEP (dr);
+ }
+
+ /* Return a value that is negative iff DR has a negative step. */
+
+ tree
+ dr_direction_indicator (struct data_reference *dr)
+ {
+ return dr_step_indicator (dr, 0);
+ }
+
+ /* Return a value that is zero iff DR has a zero step. */
+
+ tree
+ dr_zero_step_indicator (struct data_reference *dr)
+ {
+ return dr_step_indicator (dr, 1);
+ }
+
+ /* Return true if DR is known to have a nonnegative (but possibly zero)
+ step. */
+
+ bool
+ dr_known_forward_stride_p (struct data_reference *dr)
+ {
+ tree indicator = dr_direction_indicator (dr);
+ tree neg_step_val = fold_binary (LT_EXPR, boolean_type_node,
+ fold_convert (ssizetype, indicator),
+ ssize_int (0));
+ return neg_step_val && integer_zerop (neg_step_val);
+ }
Index: gcc/tree-loop-distribution.c
===================================================================
*** gcc/tree-loop-distribution.c 2017-11-17 21:44:41.257891095 +0000
--- gcc/tree-loop-distribution.c 2017-11-17 22:11:53.270650578 +0000
*************** break_alias_scc_partitions (struct graph
*** 2324,2339 ****
static tree
data_ref_segment_size (struct data_reference *dr, tree niters)
{
! tree segment_length;
!
! if (integer_zerop (DR_STEP (dr)))
! segment_length = TYPE_SIZE_UNIT (TREE_TYPE (DR_REF (dr)));
! else
! segment_length = size_binop (MULT_EXPR,
! fold_convert (sizetype, DR_STEP (dr)),
! fold_convert (sizetype, niters));
!
! return segment_length;
}
/* Return true if LOOP's latch is dominated by statement for data reference
--- 2324,2335 ----
static tree
data_ref_segment_size (struct data_reference *dr, tree niters)
{
! niters = size_binop (MINUS_EXPR,
! fold_convert (sizetype, niters),
! size_one_node);
! return size_binop (MULT_EXPR,
! fold_convert (sizetype, DR_STEP (dr)),
! fold_convert (sizetype, niters));
}
/* Return true if LOOP's latch is dominated by statement for data reference
*************** compute_alias_check_pairs (struct loop *
*** 2388,2396 ****
else
seg_length_b = data_ref_segment_size (dr_b, niters);
dr_with_seg_len_pair_t dr_with_seg_len_pair
! (dr_with_seg_len (dr_a, seg_length_a),
! dr_with_seg_len (dr_b, seg_length_b));
/* Canonicalize pairs by sorting the two DR members. */
if (comp_res > 0)
--- 2384,2399 ----
else
seg_length_b = data_ref_segment_size (dr_b, niters);
+ unsigned HOST_WIDE_INT access_size_a
+ = tree_to_uhwi (TYPE_SIZE_UNIT (TREE_TYPE (DR_REF (dr_a))));
+ unsigned HOST_WIDE_INT access_size_b
+ = tree_to_uhwi (TYPE_SIZE_UNIT (TREE_TYPE (DR_REF (dr_b))));
+ unsigned int align_a = TYPE_ALIGN_UNIT (TREE_TYPE (DR_REF (dr_a)));
+ unsigned int align_b = TYPE_ALIGN_UNIT (TREE_TYPE (DR_REF (dr_b)));
+
dr_with_seg_len_pair_t dr_with_seg_len_pair
! (dr_with_seg_len (dr_a, seg_length_a, access_size_a, align_a),
! dr_with_seg_len (dr_b, seg_length_b, access_size_b, align_b));
/* Canonicalize pairs by sorting the two DR members. */
if (comp_res > 0)
Index: gcc/tree-vect-data-refs.c
===================================================================
*** gcc/tree-vect-data-refs.c 2017-11-17 22:07:58.225016438 +0000
--- gcc/tree-vect-data-refs.c 2017-11-17 22:11:53.272510088 +0000
*************** vect_mark_for_runtime_alias_test (ddr_p
*** 168,173 ****
--- 168,217 ----
return true;
}
+ /* Record that loop LOOP_VINFO needs to check that VALUE is nonzero. */
+
+ static void
+ vect_check_nonzero_value (loop_vec_info loop_vinfo, tree value)
+ {
+ vec<tree> checks = LOOP_VINFO_CHECK_NONZERO (loop_vinfo);
+ for (unsigned int i = 0; i < checks.length(); ++i)
+ if (checks[i] == value)
+ return;
+
+ if (dump_enabled_p ())
+ {
+ dump_printf_loc (MSG_NOTE, vect_location, "need run-time check that ");
+ dump_generic_expr (MSG_NOTE, TDF_SLIM, value);
+ dump_printf (MSG_NOTE, " is nonzero\n");
+ }
+ LOOP_VINFO_CHECK_NONZERO (loop_vinfo).safe_push (value);
+ }
+
+ /* Return true if we know that the order of vectorized STMT_A and
+ vectorized STMT_B will be the same as the order of STMT_A and STMT_B.
+ At least one of the statements is a write. */
+
+ static bool
+ vect_preserves_scalar_order_p (gimple *stmt_a, gimple *stmt_b)
+ {
+ stmt_vec_info stmtinfo_a = vinfo_for_stmt (stmt_a);
+ stmt_vec_info stmtinfo_b = vinfo_for_stmt (stmt_b);
+
+ /* Single statements are always kept in their original order. */
+ if (!STMT_VINFO_GROUPED_ACCESS (stmtinfo_a)
+ && !STMT_VINFO_GROUPED_ACCESS (stmtinfo_b))
+ return true;
+
+ /* STMT_A and STMT_B belong to overlapping groups. All loads in a
+ group are emitted at the position of the first scalar load and all
+ stores in a group are emitted at the position of the last scalar store.
+ Thus writes will happen no earlier than their current position
+ (but could happen later) while reads will happen no later than their
+ current position (but could happen earlier). Reordering is therefore
+ only possible if the first access is a write. */
+ gimple *earlier_stmt = get_earlier_stmt (stmt_a, stmt_b);
+ return !DR_IS_WRITE (STMT_VINFO_DATA_REF (vinfo_for_stmt (earlier_stmt)));
+ }
/* A subroutine of vect_analyze_data_ref_dependence. Handle
DDR_COULD_BE_INDEPENDENT_P ddr DDR that has a known set of dependence
*************** vect_analyze_data_ref_dependence (struct
*** 413,434 ****
... = a[i];
a[i+1] = ...;
where loads from the group interleave with the store. */
! if (STMT_VINFO_GROUPED_ACCESS (stmtinfo_a)
! || STMT_VINFO_GROUPED_ACCESS (stmtinfo_b))
{
! gimple *earlier_stmt;
! earlier_stmt = get_earlier_stmt (DR_STMT (dra), DR_STMT (drb));
! if (DR_IS_WRITE
! (STMT_VINFO_DATA_REF (vinfo_for_stmt (earlier_stmt))))
{
if (dump_enabled_p ())
dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
! "READ_WRITE dependence in interleaving."
! "\n");
return true;
}
}
-
continue;
}
--- 457,483 ----
... = a[i];
a[i+1] = ...;
where loads from the group interleave with the store. */
! if (!vect_preserves_scalar_order_p (DR_STMT (dra), DR_STMT (drb)))
! {
! if (dump_enabled_p ())
! dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
! "READ_WRITE dependence in interleaving.\n");
! return true;
! }
!
! if (!loop->force_vectorize)
{
! tree indicator = dr_zero_step_indicator (dra);
! if (TREE_CODE (indicator) != INTEGER_CST)
! vect_check_nonzero_value (loop_vinfo, indicator);
! else if (integer_zerop (indicator))
{
if (dump_enabled_p ())
dump_printf_loc (MSG_MISSED_OPTIMIZATION, vect_location,
! "access also has a zero step\n");
return true;
}
}
continue;
}
*************** vect_analyze_data_ref_accesses (vec_info
*** 3025,3062 ****
/* Function vect_vfa_segment_size.
- Create an expression that computes the size of segment
- that will be accessed for a data reference. The functions takes into
- account that realignment loads may access one more vector.
-
Input:
DR: The data reference.
LENGTH_FACTOR: segment length to consider.
! Return an expression whose value is the size of segment which will be
! accessed by DR. */
static tree
vect_vfa_segment_size (struct data_reference *dr, tree length_factor)
{
! tree segment_length;
! if (integer_zerop (DR_STEP (dr)))
! segment_length = TYPE_SIZE_UNIT (TREE_TYPE (DR_REF (dr)));
! else
! segment_length = size_binop (MULT_EXPR,
! fold_convert (sizetype, DR_STEP (dr)),
! fold_convert (sizetype, length_factor));
! if (vect_supportable_dr_alignment (dr, false)
! == dr_explicit_realign_optimized)
{
! tree vector_size = TYPE_SIZE_UNIT
! (STMT_VINFO_VECTYPE (vinfo_for_stmt (DR_STMT (dr))));
!
! segment_length = size_binop (PLUS_EXPR, segment_length, vector_size);
}
! return segment_length;
}
/* Function vect_no_alias_p.
--- 3074,3130 ----
/* Function vect_vfa_segment_size.
Input:
DR: The data reference.
LENGTH_FACTOR: segment length to consider.
! Return a value suitable for the dr_with_seg_len::seg_len field.
! This is the "distance travelled" by the pointer from the first
! iteration in the segment to the last. Note that it does not include
! the size of the access; in effect it only describes the first byte. */
static tree
vect_vfa_segment_size (struct data_reference *dr, tree length_factor)
{
! length_factor = size_binop (MINUS_EXPR,
! fold_convert (sizetype, length_factor),
! size_one_node);
! return size_binop (MULT_EXPR, fold_convert (sizetype, DR_STEP (dr)),
! length_factor);
! }
! /* Return a value that, when added to abs (vect_vfa_segment_size (dr)),
! gives the worst-case number of bytes covered by the segment. */
! static unsigned HOST_WIDE_INT
! vect_vfa_access_size (data_reference *dr)
! {
! stmt_vec_info stmt_vinfo = vinfo_for_stmt (DR_STMT (dr));
! tree ref_type = TREE_TYPE (DR_REF (dr));
! unsigned HOST_WIDE_INT ref_size = tree_to_uhwi (TYPE_SIZE_UNIT (ref_type));
! unsigned HOST_WIDE_INT access_size = ref_size;
! if (GROUP_FIRST_ELEMENT (stmt_vinfo))
{
! gcc_assert (GROUP_FIRST_ELEMENT (stmt_vinfo) == DR_STMT (dr));
! access_size *= GROUP_SIZE (stmt_vinfo) - GROUP_GAP (stmt_vinfo);
! }
! if (STMT_VINFO_VEC_STMT (stmt_vinfo)
! && (vect_supportable_dr_alignment (dr, false)
! == dr_explicit_realign_optimized))
! {
! /* We might access a full vector's worth. */
! tree vectype = STMT_VINFO_VECTYPE (stmt_vinfo);
! access_size += tree_to_uhwi (TYPE_SIZE_UNIT (vectype)) - ref_size;
}
! return access_size;
! }
!
! /* Get the minimum alignment for all the scalar accesses that DR describes. */
!
! static unsigned int
! vect_vfa_align (const data_reference *dr)
! {
! return TYPE_ALIGN_UNIT (TREE_TYPE (DR_REF (dr)));
}
/* Function vect_no_alias_p.
*************** vect_vfa_segment_size (struct data_refer
*** 3064,3076 ****
Given data references A and B with equal base and offset, see whether
the alias relation can be decided at compilation time. Return 1 if
it can and the references alias, 0 if it can and the references do
! not alias, and -1 if we cannot decide at compile time. SEGMENT_LENGTH_A
! and SEGMENT_LENGTH_B are the memory lengths accessed by A and B
! respectively. */
static int
vect_compile_time_alias (struct data_reference *a, struct data_reference *b,
! tree segment_length_a, tree segment_length_b)
{
poly_offset_int offset_a = wi::to_poly_offset (DR_INIT (a));
poly_offset_int offset_b = wi::to_poly_offset (DR_INIT (b));
--- 3132,3146 ----
Given data references A and B with equal base and offset, see whether
the alias relation can be decided at compilation time. Return 1 if
it can and the references alias, 0 if it can and the references do
! not alias, and -1 if we cannot decide at compile time. SEGMENT_LENGTH_A,
! SEGMENT_LENGTH_B, ACCESS_SIZE_A and ACCESS_SIZE_B are the equivalent
! of dr_with_seg_len::{seg_len,access_size} for A and B. */
static int
vect_compile_time_alias (struct data_reference *a, struct data_reference *b,
! tree segment_length_a, tree segment_length_b,
! unsigned HOST_WIDE_INT access_size_a,
! unsigned HOST_WIDE_INT access_size_b)
{
poly_offset_int offset_a = wi::to_poly_offset (DR_INIT (a));
poly_offset_int offset_b = wi::to_poly_offset (DR_INIT (b));
*************** vect_compile_time_alias (struct data_ref
*** 3083,3100 ****
if (tree_int_cst_compare (DR_STEP (a), size_zero_node) < 0)
{
const_length_a = (-wi::to_poly_wide (segment_length_a)).force_uhwi ();
! offset_a = (offset_a + vect_get_scalar_dr_size (a)) - const_length_a;
}
else
const_length_a = tree_to_poly_uint64 (segment_length_a);
if (tree_int_cst_compare (DR_STEP (b), size_zero_node) < 0)
{
const_length_b = (-wi::to_poly_wide (segment_length_b)).force_uhwi ();
! offset_b = (offset_b + vect_get_scalar_dr_size (b)) - const_length_b;
}
else
const_length_b = tree_to_poly_uint64 (segment_length_b);
if (ranges_must_overlap_p (offset_a, const_length_a,
offset_b, const_length_b))
return 1;
--- 3153,3173 ----
if (tree_int_cst_compare (DR_STEP (a), size_zero_node) < 0)
{
const_length_a = (-wi::to_poly_wide (segment_length_a)).force_uhwi ();
! offset_a = (offset_a + access_size_a) - const_length_a;
}
else
const_length_a = tree_to_poly_uint64 (segment_length_a);
if (tree_int_cst_compare (DR_STEP (b), size_zero_node) < 0)
{
const_length_b = (-wi::to_poly_wide (segment_length_b)).force_uhwi ();
! offset_b = (offset_b + access_size_b) - const_length_b;
}
else
const_length_b = tree_to_poly_uint64 (segment_length_b);
+ const_length_a += access_size_a;
+ const_length_b += access_size_b;
+
if (ranges_must_overlap_p (offset_a, const_length_a,
offset_b, const_length_b))
return 1;
*************** dependence_distance_ge_vf (data_dependen
*** 3144,3149 ****
--- 3217,3324 ----
return true;
}
+ /* Dump LOWER_BOUND using flags DUMP_KIND. Dumps are known to be enabled. */
+
+ static void
+ dump_lower_bound (int dump_kind, const vec_lower_bound &lower_bound)
+ {
+ dump_printf (dump_kind, "%s (", lower_bound.unsigned_p ? "unsigned" : "abs");
+ dump_generic_expr (dump_kind, TDF_SLIM, lower_bound.expr);
+ dump_printf (dump_kind, ") >= ");
+ dump_dec (dump_kind, lower_bound.min_value);
+ }
+
+ /* Record that the vectorized loop requires the vec_lower_bound described
+ by EXPR, UNSIGNED_P and MIN_VALUE. */
+
+ static void
+ vect_check_lower_bound (loop_vec_info loop_vinfo, tree expr, bool unsigned_p,
+ poly_uint64 min_value)
+ {
+ vec<vec_lower_bound> lower_bounds = LOOP_VINFO_LOWER_BOUNDS (loop_vinfo);
+ for (unsigned int i = 0; i < lower_bounds.length (); ++i)
+ if (operand_equal_p (lower_bounds[i].expr, expr, 0))
+ {
+ unsigned_p &= lower_bounds[i].unsigned_p;
+ min_value = upper_bound (lower_bounds[i].min_value, min_value);
+ if (lower_bounds[i].unsigned_p != unsigned_p
+ || may_lt (lower_bounds[i].min_value, min_value))
+ {
+ lower_bounds[i].unsigned_p = unsigned_p;
+ lower_bounds[i].min_value = min_value;
+ if (dump_enabled_p ())
+ {
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "updating run-time check to ");
+ dump_lower_bound (MSG_NOTE, lower_bounds[i]);
+ dump_printf (MSG_NOTE, "\n");
+ }
+ }
+ return;
+ }
+
+ vec_lower_bound lower_bound (expr, unsigned_p, min_value);
+ if (dump_enabled_p ())
+ {
+ dump_printf_loc (MSG_NOTE, vect_location, "need a run-time check that ");
+ dump_lower_bound (MSG_NOTE, lower_bound);
+ dump_printf (MSG_NOTE, "\n");
+ }
+ LOOP_VINFO_LOWER_BOUNDS (loop_vinfo).safe_push (lower_bound);
+ }
+
+ /* Return true if it's unlikely that the step of the vectorized form of DR
+ will span fewer than GAP bytes. */
+
+ static bool
+ vect_small_gap_p (loop_vec_info loop_vinfo, data_reference *dr, poly_int64 gap)
+ {
+ stmt_vec_info stmt_info = vinfo_for_stmt (DR_STMT (dr));
+ HOST_WIDE_INT count
+ = estimated_poly_value (LOOP_VINFO_VECT_FACTOR (loop_vinfo));
+ if (GROUP_FIRST_ELEMENT (stmt_info))
+ count *= GROUP_SIZE (vinfo_for_stmt (GROUP_FIRST_ELEMENT (stmt_info)));
+ return estimated_poly_value (gap) <= count * vect_get_scalar_dr_size (dr);
+ }
+
+ /* Return true if we know that there is no alias between DR_A and DR_B
+ when abs (DR_STEP (DR_A)) >= N for some N. When returning true, set
+ *LOWER_BOUND_OUT to this N. */
+
+ static bool
+ vectorizable_with_step_bound_p (data_reference *dr_a, data_reference *dr_b,
+ poly_uint64 *lower_bound_out)
+ {
+ /* Check that there is a constant gap of known sign between DR_A
+ and DR_B. */
+ poly_int64 init_a, init_b;
+ if (!operand_equal_p (DR_BASE_ADDRESS (dr_a), DR_BASE_ADDRESS (dr_b), 0)
+ || !operand_equal_p (DR_OFFSET (dr_a), DR_OFFSET (dr_b), 0)
+ || !operand_equal_p (DR_STEP (dr_a), DR_STEP (dr_b), 0)
+ || !poly_int_tree_p (DR_INIT (dr_a), &init_a)
+ || !poly_int_tree_p (DR_INIT (dr_b), &init_b)
+ || !ordered_p (init_a, init_b))
+ return false;
+
+ /* Sort DR_A and DR_B by the address they access. */
+ if (may_lt (init_b, init_a))
+ {
+ std::swap (init_a, init_b);
+ std::swap (dr_a, dr_b);
+ }
+
+ /* If the two accesses could be dependent within a scalar iteration,
+ make sure that we'd retain their order. */
+ if (may_gt (init_a + vect_get_scalar_dr_size (dr_a), init_b)
+ && !vect_preserves_scalar_order_p (DR_STMT (dr_a), DR_STMT (dr_b)))
+ return false;
+
+ /* There is no alias if abs (DR_STEP) is greater than or equal to
+ the bytes spanned by the combination of the two accesses. */
+ *lower_bound_out = init_b + vect_get_scalar_dr_size (dr_b) - init_a;
+ return true;
+ }
+
/* Function vect_prune_runtime_alias_test_list.
Prune a list of ddrs to be tested at run-time by versioning for alias.
*************** vect_prune_runtime_alias_test_list (loop
*** 3173,3178 ****
--- 3348,3366 ----
dump_printf_loc (MSG_NOTE, vect_location,
"=== vect_prune_runtime_alias_test_list ===\n");
+ /* Step values are irrelevant for aliasing if the number of vector
+ iterations is equal to the number of scalar iterations (which can
+ happen for fully-SLP loops). */
+ bool ignore_step_p = must_eq (LOOP_VINFO_VECT_FACTOR (loop_vinfo), 1U);
+
+ if (!ignore_step_p)
+ {
+ /* Convert the checks for nonzero steps into bound tests. */
+ tree value;
+ FOR_EACH_VEC_ELT (LOOP_VINFO_CHECK_NONZERO (loop_vinfo), i, value)
+ vect_check_lower_bound (loop_vinfo, value, true, 1);
+ }
+
if (may_alias_ddrs.is_empty ())
return true;
*************** vect_prune_runtime_alias_test_list (loop
*** 3186,3194 ****
--- 3374,3385 ----
FOR_EACH_VEC_ELT (may_alias_ddrs, i, ddr)
{
int comp_res;
+ poly_uint64 lower_bound;
struct data_reference *dr_a, *dr_b;
gimple *dr_group_first_a, *dr_group_first_b;
tree segment_length_a, segment_length_b;
+ unsigned HOST_WIDE_INT access_size_a, access_size_b;
+ unsigned int align_a, align_b;
gimple *stmt_a, *stmt_b;
/* Ignore the alias if the VF we chose ended up being no greater
*************** vect_prune_runtime_alias_test_list (loop
*** 3216,3221 ****
--- 3407,3470 ----
dr_a = DDR_A (ddr);
stmt_a = DR_STMT (DDR_A (ddr));
+
+ dr_b = DDR_B (ddr);
+ stmt_b = DR_STMT (DDR_B (ddr));
+
+ /* Skip the pair if inter-iteration dependencies are irrelevant
+ and intra-iteration dependencies are guaranteed to be honored. */
+ if (ignore_step_p
+ && (vect_preserves_scalar_order_p (stmt_a, stmt_b)
+ || vectorizable_with_step_bound_p (dr_a, dr_b, &lower_bound)))
+ {
+ if (dump_enabled_p ())
+ {
+ dump_printf_loc (MSG_NOTE, vect_location,
+ "no need for alias check between ");
+ dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_a));
+ dump_printf (MSG_NOTE, " and ");
+ dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_b));
+ dump_printf (MSG_NOTE, " when VF is 1\n");
+ }
+ continue;
+ }
+
+ /* See whether we can handle the alias using a bounds check on
+ the step, and whether that's likely to be the best approach.
+ (It might not be, for example, if the minimum step is much larger
+ than the number of bytes handled by one vector iteration.) */
+ if (!ignore_step_p
+ && TREE_CODE (DR_STEP (dr_a)) != INTEGER_CST
+ && vectorizable_with_step_bound_p (dr_a, dr_b, &lower_bound)
+ && (vect_small_gap_p (loop_vinfo, dr_a, lower_bound)
+ || vect_small_gap_p (loop_vinfo, dr_b, lower_bound)))
+ {
+ bool unsigned_p = dr_known_forward_stride_p (dr_a);
+ if (dump_enabled_p ())
+ {
+ dump_printf_loc (MSG_NOTE, vect_location, "no alias between ");
+ dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_a));
+ dump_printf (MSG_NOTE, " and ");
+ dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_b));
+ dump_printf (MSG_NOTE, " when the step ");
+ dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_STEP (dr_a));
+ dump_printf (MSG_NOTE, " is outside ");
+ if (unsigned_p)
+ dump_printf (MSG_NOTE, "[0");
+ else
+ {
+ dump_printf (MSG_NOTE, "(");
+ dump_dec (MSG_NOTE, poly_int64 (-lower_bound));
+ }
+ dump_printf (MSG_NOTE, ", ");
+ dump_dec (MSG_NOTE, lower_bound);
+ dump_printf (MSG_NOTE, ")\n");
+ }
+ vect_check_lower_bound (loop_vinfo, DR_STEP (dr_a), unsigned_p,
+ lower_bound);
+ continue;
+ }
+
dr_group_first_a = GROUP_FIRST_ELEMENT (vinfo_for_stmt (stmt_a));
if (dr_group_first_a)
{
*************** vect_prune_runtime_alias_test_list (loop
*** 3223,3230 ****
dr_a = STMT_VINFO_DATA_REF (vinfo_for_stmt (stmt_a));
}
- dr_b = DDR_B (ddr);
- stmt_b = DR_STMT (DDR_B (ddr));
dr_group_first_b = GROUP_FIRST_ELEMENT (vinfo_for_stmt (stmt_b));
if (dr_group_first_b)
{
--- 3472,3477 ----
*************** vect_prune_runtime_alias_test_list (loop
*** 3232,3243 ****
dr_b = STMT_VINFO_DATA_REF (vinfo_for_stmt (stmt_b));
}
! if (!operand_equal_p (DR_STEP (dr_a), DR_STEP (dr_b), 0))
! length_factor = scalar_loop_iters;
else
! length_factor = size_int (vect_factor);
! segment_length_a = vect_vfa_segment_size (dr_a, length_factor);
! segment_length_b = vect_vfa_segment_size (dr_b, length_factor);
comp_res = data_ref_compare_tree (DR_BASE_ADDRESS (dr_a),
DR_BASE_ADDRESS (dr_b));
--- 3479,3502 ----
dr_b = STMT_VINFO_DATA_REF (vinfo_for_stmt (stmt_b));
}
! if (ignore_step_p)
! {
! segment_length_a = size_zero_node;
! segment_length_b = size_zero_node;
! }
else
! {
! if (!operand_equal_p (DR_STEP (dr_a), DR_STEP (dr_b), 0))
! length_factor = scalar_loop_iters;
! else
! length_factor = size_int (vect_factor);
! segment_length_a = vect_vfa_segment_size (dr_a, length_factor);
! segment_length_b = vect_vfa_segment_size (dr_b, length_factor);
! }
! access_size_a = vect_vfa_access_size (dr_a);
! access_size_b = vect_vfa_access_size (dr_b);
! align_a = vect_vfa_align (dr_a);
! align_b = vect_vfa_align (dr_b);
comp_res = data_ref_compare_tree (DR_BASE_ADDRESS (dr_a),
DR_BASE_ADDRESS (dr_b));
*************** vect_prune_runtime_alias_test_list (loop
*** 3254,3260 ****
{
int res = vect_compile_time_alias (dr_a, dr_b,
segment_length_a,
! segment_length_b);
if (res == 0)
continue;
--- 3513,3534 ----
{
int res = vect_compile_time_alias (dr_a, dr_b,
segment_length_a,
! segment_length_b,
! access_size_a,
! access_size_b);
! if (res >= 0 && dump_enabled_p ())
! {
! dump_printf_loc (MSG_NOTE, vect_location,
! "can tell at compile time that ");
! dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_a));
! dump_printf (MSG_NOTE, " and ");
! dump_generic_expr (MSG_NOTE, TDF_SLIM, DR_REF (dr_b));
! if (res == 0)
! dump_printf (MSG_NOTE, " do not alias\n");
! else
! dump_printf (MSG_NOTE, " alias\n");
! }
!
if (res == 0)
continue;
*************** vect_prune_runtime_alias_test_list (loop
*** 3268,3275 ****
}
dr_with_seg_len_pair_t dr_with_seg_len_pair
! (dr_with_seg_len (dr_a, segment_length_a),
! dr_with_seg_len (dr_b, segment_length_b));
/* Canonicalize pairs by sorting the two DR members. */
if (comp_res > 0)
--- 3542,3549 ----
}
dr_with_seg_len_pair_t dr_with_seg_len_pair
! (dr_with_seg_len (dr_a, segment_length_a, access_size_a, align_a),
! dr_with_seg_len (dr_b, segment_length_b, access_size_b, align_b));
/* Canonicalize pairs by sorting the two DR members. */
if (comp_res > 0)
*************** vect_prune_runtime_alias_test_list (loop
*** 3282,3287 ****
--- 3556,3562 ----
unsigned int count = (comp_alias_ddrs.length ()
+ check_unequal_addrs.length ());
+
dump_printf_loc (MSG_NOTE, vect_location,
"improved number of alias checks from %d to %d\n",
may_alias_ddrs.length (), count);
Index: gcc/tree-vect-loop-manip.c
===================================================================
*** gcc/tree-vect-loop-manip.c 2017-11-17 21:48:20.590673490 +0000
--- gcc/tree-vect-loop-manip.c 2017-11-17 22:11:53.272510088 +0000
*************** vect_create_cond_for_unequal_addrs (loop
*** 2851,2856 ****
--- 2851,2881 ----
}
}
+ /* Create an expression that is true when all lower-bound conditions for
+ the vectorized loop are met. Chain this condition with *COND_EXPR. */
+
+ static void
+ vect_create_cond_for_lower_bounds (loop_vec_info loop_vinfo, tree *cond_expr)
+ {
+ vec<vec_lower_bound> lower_bounds = LOOP_VINFO_LOWER_BOUNDS (loop_vinfo);
+ for (unsigned int i = 0; i < lower_bounds.length (); ++i)
+ {
+ tree expr = lower_bounds[i].expr;
+ tree type = unsigned_type_for (TREE_TYPE (expr));
+ expr = fold_convert (type, expr);
+ poly_uint64 bound = lower_bounds[i].min_value;
+ if (!lower_bounds[i].unsigned_p)
+ {
+ expr = fold_build2 (PLUS_EXPR, type, expr,
+ build_int_cstu (type, bound - 1));
+ bound += bound - 1;
+ }
+ tree part_cond_expr = fold_build2 (GE_EXPR, boolean_type_node, expr,
+ build_int_cstu (type, bound));
+ chain_cond_expr (cond_expr, part_cond_expr);
+ }
+ }
+
/* Function vect_create_cond_for_alias_checks.
Create a conditional expression that represents the run-time checks for
*************** vect_loop_versioning (loop_vec_info loop
*** 2962,2967 ****
--- 2987,2993 ----
if (version_alias)
{
vect_create_cond_for_unequal_addrs (loop_vinfo, &cond_expr);
+ vect_create_cond_for_lower_bounds (loop_vinfo, &cond_expr);
vect_create_cond_for_alias_checks (loop_vinfo, &cond_expr);
}
Index: gcc/tree-vect-loop.c
===================================================================
*** gcc/tree-vect-loop.c 2017-11-17 21:48:20.646463984 +0000
--- gcc/tree-vect-loop.c 2017-11-17 22:11:53.273439843 +0000
*************** vect_analyze_loop_2 (loop_vec_info loop_
*** 2472,2477 ****
--- 2472,2478 ----
}
}
/* Free optimized alias test DDRS. */
+ LOOP_VINFO_LOWER_BOUNDS (loop_vinfo).truncate (0);
LOOP_VINFO_COMP_ALIAS_DDRS (loop_vinfo).release ();
LOOP_VINFO_CHECK_UNEQUAL_ADDRS (loop_vinfo).release ();
/* Reset target cost data. */
*************** vect_estimate_min_profitable_iters (loop
*** 3668,3673 ****
--- 3669,3686 ----
/* Count LEN - 1 ANDs and LEN comparisons. */
(void) add_stmt_cost (target_cost_data, len * 2 - 1, scalar_stmt,
NULL, 0, vect_prologue);
+ len = LOOP_VINFO_LOWER_BOUNDS (loop_vinfo).length ();
+ if (len)
+ {
+ /* Count LEN - 1 ANDs and LEN comparisons. */
+ unsigned int nstmts = len * 2 - 1;
+ /* +1 for each bias that needs adding. */
+ for (unsigned int i = 0; i < len; ++i)
+ if (!LOOP_VINFO_LOWER_BOUNDS (loop_vinfo)[i].unsigned_p)
+ nstmts += 1;
+ (void) add_stmt_cost (target_cost_data, nstmts, scalar_stmt,
+ NULL, 0, vect_prologue);
+ }
dump_printf (MSG_NOTE,
"cost model: Adding cost of checks for loop "
"versioning aliasing.\n");
Index: gcc/testsuite/gcc.dg/vect/bb-slp-cond-1.c
===================================================================
*** gcc/testsuite/gcc.dg/vect/bb-slp-cond-1.c 2017-11-09 15:15:06.847657980 +0000
--- gcc/testsuite/gcc.dg/vect/bb-slp-cond-1.c 2017-11-17 22:11:53.266001803 +0000
*************** int main ()
*** 45,54 ****
return 0;
}
! /* Basic blocks of if-converted loops are vectorized from within the loop
! vectorizer pass. In this case it is really a deficiency in loop
! vectorization data dependence analysis that causes us to require
! basic block vectorization in the first place. */
!
! /* { dg-final { scan-tree-dump-times "basic block vectorized" 1 "vect" { target vect_element_align } } } */
--- 45,50 ----
return 0;
}
! /* { dg-final { scan-tree-dump {(no need for alias check [^\n]* when VF is 1|no alias between [^\n]* when [^\n]* is outside \(-16, 16\))} "vect" { target vect_element_align } } } */
! /* { dg-final { scan-tree-dump-times "loop vectorized" 1 "vect" { target vect_element_align } } } */
Index: gcc/testsuite/gcc.dg/vect/vect-alias-check-10.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.dg/vect/vect-alias-check-10.c 2017-11-17 22:11:53.266001803 +0000
***************
*** 0 ****
--- 1,69 ----
+ /* { dg-do run } */
+
+ #define N 87
+ #define M 6
+
+ typedef signed char sc;
+ typedef unsigned char uc;
+ typedef signed short ss;
+ typedef unsigned short us;
+ typedef int si;
+ typedef unsigned int ui;
+ typedef signed long long sll;
+ typedef unsigned long long ull;
+
+ #define FOR_EACH_TYPE(M) \
+ M (sc) M (uc) \
+ M (ss) M (us) \
+ M (si) M (ui) \
+ M (sll) M (ull) \
+ M (float) M (double)
+
+ #define TEST_VALUE(I) ((I) * 5 / 2)
+
+ #define ADD_TEST(TYPE) \
+ void __attribute__((noinline, noclone)) \
+ test_##TYPE (TYPE *a, int step) \
+ { \
+ for (int i = 0; i < N; ++i) \
+ { \
+ a[i * step + 0] = a[i * step + 0] + 1; \
+ a[i * step + 1] = a[i * step + 1] + 2; \
+ a[i * step + 2] = a[i * step + 2] + 4; \
+ a[i * step + 3] = a[i * step + 3] + 8; \
+ } \
+ } \
+ void __attribute__((noinline, noclone)) \
+ ref_##TYPE (TYPE *a, int step) \
+ { \
+ for (int i = 0; i < N; ++i) \
+ { \
+ a[i * step + 0] = a[i * step + 0] + 1; \
+ a[i * step + 1] = a[i * step + 1] + 2; \
+ a[i * step + 2] = a[i * step + 2] + 4; \
+ a[i * step + 3] = a[i * step + 3] + 8; \
+ asm volatile (""); \
+ } \
+ }
+
+ #define DO_TEST(TYPE) \
+ for (int j = -M; j <= M; ++j) \
+ { \
+ TYPE a[N * M], b[N * M]; \
+ for (int i = 0; i < N * M; ++i) \
+ a[i] = b[i] = TEST_VALUE (i); \
+ int offset = (j < 0 ? N * M - 4 : 0); \
+ test_##TYPE (a + offset, j); \
+ ref_##TYPE (b + offset, j); \
+ if (__builtin_memcmp (a, b, sizeof (a)) != 0) \
+ __builtin_abort (); \
+ }
+
+ FOR_EACH_TYPE (ADD_TEST)
+
+ int
+ main (void)
+ {
+ FOR_EACH_TYPE (DO_TEST)
+ return 0;
+ }
Index: gcc/testsuite/gcc.dg/vect/vect-alias-check-11.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.dg/vect/vect-alias-check-11.c 2017-11-17 22:11:53.266001803 +0000
***************
*** 0 ****
--- 1,99 ----
+ /* { dg-do run } */
+
+ #define N 87
+ #define M 6
+
+ typedef signed char sc;
+ typedef unsigned char uc;
+ typedef signed short ss;
+ typedef unsigned short us;
+ typedef int si;
+ typedef unsigned int ui;
+ typedef signed long long sll;
+ typedef unsigned long long ull;
+
+ #define FOR_EACH_TYPE(M) \
+ M (sc) M (uc) \
+ M (ss) M (us) \
+ M (si) M (ui) \
+ M (sll) M (ull) \
+ M (float) M (double)
+
+ #define TEST_VALUE1(I) ((I) * 5 / 2)
+ #define TEST_VALUE2(I) ((I) * 11 / 5)
+
+ #define ADD_TEST(TYPE) \
+ void __attribute__((noinline, noclone)) \
+ test_##TYPE (TYPE *restrict a, TYPE *restrict b, \
+ int step) \
+ { \
+ for (int i = 0; i < N; ++i) \
+ { \
+ TYPE r1 = a[i * step + 0] += 1; \
+ a[i * step + 1] += 2; \
+ a[i * step + 2] += 4; \
+ a[i * step + 3] += 8; \
+ b[i] += r1; \
+ } \
+ } \
+ \
+ void __attribute__((noinline, noclone)) \
+ ref_##TYPE (TYPE *restrict a, TYPE *restrict b, \
+ int step) \
+ { \
+ for (int i = 0; i < N; ++i) \
+ { \
+ TYPE r1 = a[i * step + 0] += 1; \
+ a[i * step + 1] += 2; \
+ a[i * step + 2] += 4; \
+ a[i * step + 3] += 8; \
+ b[i] += r1; \
+ asm volatile (""); \
+ } \
+ }
+
+ #define DO_TEST(TYPE) \
+ for (int j = -M; j <= M; ++j) \
+ { \
+ TYPE a1[N * M], a2[N * M], b1[N], b2[N]; \
+ for (int i = 0; i < N * M; ++i) \
+ a1[i] = a2[i] = TEST_VALUE1 (i); \
+ for (int i = 0; i < N; ++i) \
+ b1[i] = b2[i] = TEST_VALUE2 (i); \
+ int offset = (j < 0 ? N * M - 4 : 0); \
+ test_##TYPE (a1 + offset, b1, j); \
+ ref_##TYPE (a2 + offset, b2, j); \
+ if (__builtin_memcmp (a1, a2, sizeof (a1)) != 0) \
+ __builtin_abort (); \
+ if (__builtin_memcmp (b1, b2, sizeof (b1)) != 0) \
+ __builtin_abort (); \
+ }
+
+ FOR_EACH_TYPE (ADD_TEST)
+
+ int
+ main (void)
+ {
+ FOR_EACH_TYPE (DO_TEST)
+ return 0;
+ }
+
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* step[^ ]* is outside \(-2, 2\)} "vect" { target vect_int } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* step[^ ]* is outside \(-3, 3\)} "vect" { target vect_int } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* step[^ ]* is outside \(-4, 4\)} "vect" { target vect_int } } } */
+ /* { dg-final { scan-tree-dump {run-time check [^\n]* abs \([^*]*\) >= 4} "vect" { target vect_int } } } */
+
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* step[^ ]* \* 2[)]* is outside \(-4, 4\)} "vect" { target vect_int } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* step[^ ]* \* 2[)]* is outside \(-6, 6\)} "vect" { target vect_int } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* step[^ ]* \* 2[)]* is outside \(-8, 8\)} "vect" { target vect_int } } } */
+ /* { dg-final { scan-tree-dump {run-time check [^\n]* abs \([^*]* \* 2[)]* >= 8} "vect" { target vect_int } } } */
+
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* step[^ ]* \* 4[)]* is outside \(-8, 8\)} "vect" { target { vect_int || vect_float } } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* step[^ ]* \* 4[)]* is outside \(-12, 12\)} "vect" { target { vect_int || vect_float } } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* step[^ ]* \* 4[)]* is outside \(-16, 16\)} "vect" { target { vect_int || vect_float } } } } */
+ /* { dg-final { scan-tree-dump {run-time check [^\n]* abs \([^*]* \* 4[)]* >= 16} "vect" { target { vect_int || vect_float } } } } */
+
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* step[^ ]* \* 8[)]* is outside \(-16, 16\)} "vect" { target vect_double } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* step[^ ]* \* 8[)]* is outside \(-24, 24\)} "vect" { target vect_double } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* step[^ ]* \* 8[)]* is outside \(-32, 32\)} "vect" { target vect_double } } } */
+ /* { dg-final { scan-tree-dump {run-time check [^\n]* abs \([^*]* \* 8[)]* >= 32} "vect" { target vect_double } } } */
Index: gcc/testsuite/gcc.dg/vect/vect-alias-check-12.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.dg/vect/vect-alias-check-12.c 2017-11-17 22:11:53.266001803 +0000
***************
*** 0 ****
--- 1,99 ----
+ /* { dg-do run } */
+
+ #define N 87
+ #define M 7
+
+ typedef signed char sc;
+ typedef unsigned char uc;
+ typedef signed short ss;
+ typedef unsigned short us;
+ typedef int si;
+ typedef unsigned int ui;
+ typedef signed long long sll;
+ typedef unsigned long long ull;
+
+ #define FOR_EACH_TYPE(M) \
+ M (sc) M (uc) \
+ M (ss) M (us) \
+ M (si) M (ui) \
+ M (sll) M (ull) \
+ M (float) M (double)
+
+ #define TEST_VALUE1(I) ((I) * 5 / 2)
+ #define TEST_VALUE2(I) ((I) * 11 / 5)
+
+ #define ADD_TEST(TYPE) \
+ void __attribute__((noinline, noclone)) \
+ test_##TYPE (TYPE *restrict a, TYPE *restrict b, \
+ int step) \
+ { \
+ step = step & M; \
+ for (int i = 0; i < N; ++i) \
+ { \
+ TYPE r1 = a[i * step + 0] += 1; \
+ a[i * step + 1] += 2; \
+ a[i * step + 2] += 4; \
+ a[i * step + 3] += 8; \
+ b[i] += r1; \
+ } \
+ } \
+ \
+ void __attribute__((noinline, noclone)) \
+ ref_##TYPE (TYPE *restrict a, TYPE *restrict b, \
+ int step) \
+ { \
+ for (unsigned short i = 0; i < N; ++i) \
+ { \
+ TYPE r1 = a[i * step + 0] += 1; \
+ a[i * step + 1] += 2; \
+ a[i * step + 2] += 4; \
+ a[i * step + 3] += 8; \
+ b[i] += r1; \
+ asm volatile (""); \
+ } \
+ }
+
+ #define DO_TEST(TYPE) \
+ for (int j = 0; j <= M; ++j) \
+ { \
+ TYPE a1[N * M], a2[N * M], b1[N], b2[N]; \
+ for (int i = 0; i < N * M; ++i) \
+ a1[i] = a2[i] = TEST_VALUE1 (i); \
+ for (int i = 0; i < N; ++i) \
+ b1[i] = b2[i] = TEST_VALUE2 (i); \
+ test_##TYPE (a1, b1, j); \
+ ref_##TYPE (a2, b2, j); \
+ if (__builtin_memcmp (a1, a2, sizeof (a1)) != 0) \
+ __builtin_abort (); \
+ if (__builtin_memcmp (b1, b2, sizeof (b1)) != 0) \
+ __builtin_abort (); \
+ }
+
+ FOR_EACH_TYPE (ADD_TEST)
+
+ int
+ main (void)
+ {
+ FOR_EACH_TYPE (DO_TEST)
+ return 0;
+ }
+
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* [_a-z][^ ]* is outside \[0, 2\)} "vect" { target vect_int } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* [_a-z][^ ]* is outside \[0, 3\)} "vect" { target vect_int } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* [_a-z][^ ]* is outside \[0, 4\)} "vect" { target vect_int } } } */
+ /* { dg-final { scan-tree-dump {run-time check [^\n]* unsigned \([^*]*\) >= 4} "vect" { target vect_int } } } */
+
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* [_a-z][^ ]* \* 2[)]* is outside \[0, 4\)} "vect" { target vect_int } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* [_a-z][^ ]* \* 2[)]* is outside \[0, 6\)} "vect" { target vect_int } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* [_a-z][^ ]* \* 2[)]* is outside \[0, 8\)} "vect" { target vect_int } } } */
+ /* { dg-final { scan-tree-dump {run-time check [^\n]* unsigned \([^*]* \* 2[)]* >= 8} "vect" { target vect_int } } } */
+
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* [_a-z][^ ]* \* 4[)]* is outside \[0, 8\)} "vect" { target { vect_int || vect_float } }} } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* [_a-z][^ ]* \* 4[)]* is outside \[0, 12\)} "vect" { target { vect_int || vect_float } }} } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* [_a-z][^ ]* \* 4[)]* is outside \[0, 16\)} "vect" { target { vect_int || vect_float } }} } */
+ /* { dg-final { scan-tree-dump {run-time check [^\n]* unsigned \([^*]* \* 4[)]* >= 16} "vect" { target { vect_int || vect_float } }} } */
+
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* [_a-z][^ ]* \* 8[)]* is outside \[0, 16\)} "vect" { target vect_double } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* [_a-z][^ ]* \* 8[)]* is outside \[0, 24\)} "vect" { target vect_double } } } */
+ /* { dg-final { scan-tree-dump {no alias between [^\n]* when [^\n]* [_a-z][^ ]* \* 8[)]* is outside \[0, 32\)} "vect" { target vect_double } } } */
+ /* { dg-final { scan-tree-dump {run-time check [^\n]* unsigned \([^*]* \* 8[)]* >= 32} "vect" { target vect_double } } } */
Index: gcc/testsuite/gcc.dg/vect/vect-alias-check-8.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.dg/vect/vect-alias-check-8.c 2017-11-17 22:11:53.266001803 +0000
***************
*** 0 ****
--- 1,62 ----
+ /* { dg-do run } */
+
+ #define N 200
+ #define DIST 32
+
+ typedef signed char sc;
+ typedef unsigned char uc;
+ typedef signed short ss;
+ typedef unsigned short us;
+ typedef int si;
+ typedef unsigned int ui;
+ typedef signed long long sll;
+ typedef unsigned long long ull;
+
+ #define FOR_EACH_TYPE(M) \
+ M (sc) M (uc) \
+ M (ss) M (us) \
+ M (si) M (ui) \
+ M (sll) M (ull) \
+ M (float) M (double)
+
+ #define TEST_VALUE(I) ((I) * 5 / 2)
+
+ #define ADD_TEST(TYPE) \
+ TYPE a_##TYPE[N * 2]; \
+ void __attribute__((noinline, noclone)) \
+ test_##TYPE (int x, int y) \
+ { \
+ for (int i = 0; i < N; ++i) \
+ a_##TYPE[i + x] += a_##TYPE[i + y]; \
+ }
+
+ #define DO_TEST(TYPE) \
+ for (int i = 0; i < DIST * 2; ++i) \
+ { \
+ for (int j = 0; j < N + DIST * 2; ++j) \
+ a_##TYPE[j] = TEST_VALUE (j); \
+ test_##TYPE (i, DIST); \
+ for (int j = 0; j < N + DIST * 2; ++j) \
+ { \
+ TYPE expected; \
+ if (j < i || j >= i + N) \
+ expected = TEST_VALUE (j); \
+ else if (i <= DIST) \
+ expected = ((TYPE) TEST_VALUE (j) \
+ + (TYPE) TEST_VALUE (j - i + DIST)); \
+ else \
+ expected = ((TYPE) TEST_VALUE (j) \
+ + a_##TYPE[j - i + DIST]); \
+ if (expected != a_##TYPE[j]) \
+ __builtin_abort (); \
+ } \
+ }
+
+ FOR_EACH_TYPE (ADD_TEST)
+
+ int
+ main (void)
+ {
+ FOR_EACH_TYPE (DO_TEST)
+ return 0;
+ }
Index: gcc/testsuite/gcc.dg/vect/vect-alias-check-9.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.dg/vect/vect-alias-check-9.c 2017-11-17 22:11:53.266001803 +0000
***************
*** 0 ****
--- 1,55 ----
+ /* { dg-do run } */
+
+ #define N 200
+ #define M 4
+
+ typedef signed char sc;
+ typedef unsigned char uc;
+ typedef signed short ss;
+ typedef unsigned short us;
+ typedef int si;
+ typedef unsigned int ui;
+ typedef signed long long sll;
+ typedef unsigned long long ull;
+
+ #define FOR_EACH_TYPE(M) \
+ M (sc) M (uc) \
+ M (ss) M (us) \
+ M (si) M (ui) \
+ M (sll) M (ull) \
+ M (float) M (double)
+
+ #define TEST_VALUE(I) ((I) * 5 / 2)
+
+ #define ADD_TEST(TYPE) \
+ void __attribute__((noinline, noclone)) \
+ test_##TYPE (TYPE *a, TYPE *b) \
+ { \
+ for (int i = 0; i < N; i += 2) \
+ { \
+ a[i + 0] = b[i + 0] + 2; \
+ a[i + 1] = b[i + 1] + 3; \
+ } \
+ }
+
+ #define DO_TEST(TYPE) \
+ for (int j = 1; j < M; ++j) \
+ { \
+ TYPE a[N + M]; \
+ for (int i = 0; i < N + M; ++i) \
+ a[i] = TEST_VALUE (i); \
+ test_##TYPE (a + j, a); \
+ for (int i = 0; i < N; i += 2) \
+ if (a[i + j] != (TYPE) (a[i] + 2) \
+ || a[i + j + 1] != (TYPE) (a[i + 1] + 3)) \
+ __builtin_abort (); \
+ }
+
+ FOR_EACH_TYPE (ADD_TEST)
+
+ int
+ main (void)
+ {
+ FOR_EACH_TYPE (DO_TEST)
+ return 0;
+ }
Index: gcc/testsuite/gcc.target/aarch64/sve_strided_load_8.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_strided_load_8.c 2017-11-17 22:11:53.266001803 +0000
***************
*** 0 ****
--- 1,15 ----
+ /* { dg-do compile } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ void
+ foo (double *x, int m)
+ {
+ for (int i = 0; i < 256; ++i)
+ x[i * m] += x[i * m];
+ }
+
+ /* { dg-final { scan-assembler-times {\tcbz\tw1,} 1 } } */
+ /* { dg-final { scan-assembler-times {\tld1d\tz[0-9]+\.d, } 1 } } */
+ /* { dg-final { scan-assembler-times {\tst1d\tz[0-9]+\.d, } 1 } } */
+ /* { dg-final { scan-assembler-times {\tldr\t} 1 } } */
+ /* { dg-final { scan-assembler-times {\tstr\t} 1 } } */
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_1.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_1.c 2017-11-17 22:11:53.266931558 +0000
***************
*** 0 ****
--- 1,27 ----
+ /* { dg-do compile } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #define TYPE int
+ #define SIZE 257
+
+ void __attribute__ ((weak))
+ f (TYPE *x, TYPE *y, unsigned short n, long m __attribute__((unused)))
+ {
+ for (int i = 0; i < SIZE; ++i)
+ x[i * n] += y[i * n];
+ }
+
+ /* { dg-final { scan-assembler {\tld1w\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tst1w\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tldr\tw[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tstr\tw[0-9]+} } } */
+ /* Should multiply by (VF-1)*4 rather than (257-1)*4. */
+ /* { dg-final { scan-assembler-not {, 1024} } } */
+ /* { dg-final { scan-assembler-not {\t.bfiz\t} } } */
+ /* { dg-final { scan-assembler-not {lsl[^\n]*[, ]10} } } */
+ /* { dg-final { scan-assembler-not {\tcmp\tx[0-9]+, 0} } } */
+ /* { dg-final { scan-assembler-not {\tcmp\tw[0-9]+, 0} } } */
+ /* { dg-final { scan-assembler-not {\tcsel\tx[0-9]+} } } */
+ /* Two range checks and a check for n being zero. */
+ /* { dg-final { scan-assembler-times {\tcmp\t} 1 } } */
+ /* { dg-final { scan-assembler-times {\tccmp\t} 2 } } */
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_1.h
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_1.h 2017-11-17 22:11:53.266931558 +0000
***************
*** 0 ****
--- 1,61 ----
+ extern void abort (void) __attribute__ ((noreturn));
+
+ #define MARGIN 6
+
+ void __attribute__ ((weak, optimize ("no-tree-vectorize")))
+ test (int n, int m, int offset)
+ {
+ int abs_n = (n < 0 ? -n : n);
+ int abs_m = (m < 0 ? -m : m);
+ int max_i = (abs_n > abs_m ? abs_n : abs_m);
+ int abs_offset = (offset < 0 ? -offset : offset);
+ int size = MARGIN * 2 + max_i * SIZE + abs_offset;
+ TYPE *array = (TYPE *) __builtin_alloca (size * sizeof (TYPE));
+ for (int i = 0; i < size; ++i)
+ array[i] = i;
+ int base_x = offset < 0 ? MARGIN - offset : MARGIN;
+ int base_y = offset < 0 ? MARGIN : MARGIN + offset;
+ int start_x = n < 0 ? base_x - n * (SIZE - 1) : base_x;
+ int start_y = m < 0 ? base_y - m * (SIZE - 1) : base_y;
+ f (&array[start_x], &array[start_y], n, m);
+ int j = 0;
+ int start = (n < 0 ? size - 1 : 0);
+ int end = (n < 0 ? -1 : size);
+ int inc = (n < 0 ? -1 : 1);
+ for (int i = start; i != end; i += inc)
+ {
+ if (j == SIZE || i != start_x + j * n)
+ {
+ if (array[i] != i)
+ abort ();
+ }
+ else if (n == 0)
+ {
+ TYPE sum = i;
+ for (; j < SIZE; j++)
+ {
+ int next_y = start_y + j * m;
+ if (n >= 0 ? next_y < i : next_y > i)
+ sum += array[next_y];
+ else if (next_y == i)
+ sum += sum;
+ else
+ sum += next_y;
+ }
+ if (array[i] != sum)
+ abort ();
+ }
+ else
+ {
+ int next_y = start_y + j * m;
+ TYPE base = i;
+ if (n >= 0 ? next_y < i : next_y > i)
+ base += array[next_y];
+ else
+ base += next_y;
+ if (array[i] != base)
+ abort ();
+ j += 1;
+ }
+ }
+ }
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_1_run.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_1_run.c 2017-11-17 22:11:53.266931558 +0000
***************
*** 0 ****
--- 1,14 ----
+ /* { dg-do run { target { aarch64_sve_hw } } } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #include "sve_var_stride_1.c"
+ #include "sve_var_stride_1.h"
+
+ int
+ main (void)
+ {
+ for (int n = 0; n < 10; ++n)
+ for (int offset = -33; offset <= 33; ++offset)
+ test (n, n, offset);
+ return 0;
+ }
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_2.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_2.c 2017-11-17 22:11:53.266931558 +0000
***************
*** 0 ****
--- 1,25 ----
+ /* { dg-do compile } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #define TYPE int
+ #define SIZE 257
+
+ void __attribute__ ((weak))
+ f (TYPE *x, TYPE *y, unsigned short n, unsigned short m)
+ {
+ for (int i = 0; i < SIZE; ++i)
+ x[i * n] += y[i * m];
+ }
+
+ /* { dg-final { scan-assembler {\tld1w\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tst1w\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tldr\tw[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tstr\tw[0-9]+} } } */
+ /* Should multiply by (257-1)*4 rather than (VF-1)*4. */
+ /* { dg-final { scan-assembler-times {\tubfiz\tx[0-9]+, x[0-9]+, 10, 16} 2 } } */
+ /* { dg-final { scan-assembler-not {\tcmp\tx[0-9]+, 0} } } */
+ /* { dg-final { scan-assembler-not {\tcmp\tw[0-9]+, 0} } } */
+ /* { dg-final { scan-assembler-not {\tcsel\tx[0-9]+} } } */
+ /* Two range checks and a check for n being zero. (m being zero is OK.) */
+ /* { dg-final { scan-assembler-times {\tcmp\t} 1 } } */
+ /* { dg-final { scan-assembler-times {\tccmp\t} 2 } } */
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_2_run.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_2_run.c 2017-11-17 22:11:53.266931558 +0000
***************
*** 0 ****
--- 1,18 ----
+ /* { dg-do run { target { aarch64_sve_hw } } } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #include "sve_var_stride_2.c"
+ #include "sve_var_stride_1.h"
+
+ int
+ main (void)
+ {
+ for (int n = 0; n < 10; ++n)
+ for (int m = 0; m < 10; ++m)
+ for (int offset = -17; offset <= 17; ++offset)
+ {
+ test (n, m, offset);
+ test (n, m, offset + n * (SIZE - 1));
+ }
+ return 0;
+ }
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_3.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_3.c 2017-11-17 22:11:53.266931558 +0000
***************
*** 0 ****
--- 1,27 ----
+ /* { dg-do compile } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #define TYPE int
+ #define SIZE 257
+
+ void __attribute__ ((weak))
+ f (TYPE *x, TYPE *y, int n, long m __attribute__((unused)))
+ {
+ for (int i = 0; i < SIZE; ++i)
+ x[i * n] += y[i * n];
+ }
+
+ /* { dg-final { scan-assembler {\tld1w\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tst1w\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tldr\tw[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tstr\tw[0-9]+} } } */
+ /* Should multiply by (VF-1)*4 rather than (257-1)*4. */
+ /* { dg-final { scan-assembler-not {, 1024} } } */
+ /* { dg-final { scan-assembler-not {\t.bfiz\t} } } */
+ /* { dg-final { scan-assembler-not {lsl[^\n]*[, ]10} } } */
+ /* { dg-final { scan-assembler-not {\tcmp\tx[0-9]+, 0} } } */
+ /* { dg-final { scan-assembler {\tcmp\tw2, 0} } } */
+ /* { dg-final { scan-assembler-times {\tcsel\tx[0-9]+} 2 } } */
+ /* Two range checks and a check for n being zero. */
+ /* { dg-final { scan-assembler {\tcmp\t} } } */
+ /* { dg-final { scan-assembler-times {\tccmp\t} 2 } } */
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_3_run.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_3_run.c 2017-11-17 22:11:53.266931558 +0000
***************
*** 0 ****
--- 1,14 ----
+ /* { dg-do run { target { aarch64_sve_hw } } } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #include "sve_var_stride_3.c"
+ #include "sve_var_stride_1.h"
+
+ int
+ main (void)
+ {
+ for (int n = -10; n < 10; ++n)
+ for (int offset = -33; offset <= 33; ++offset)
+ test (n, n, offset);
+ return 0;
+ }
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_4.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_4.c 2017-11-17 22:11:53.266931558 +0000
***************
*** 0 ****
--- 1,25 ----
+ /* { dg-do compile } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #define TYPE int
+ #define SIZE 257
+
+ void __attribute__ ((weak))
+ f (TYPE *x, TYPE *y, int n, int m)
+ {
+ for (int i = 0; i < SIZE; ++i)
+ x[i * n] += y[i * m];
+ }
+
+ /* { dg-final { scan-assembler {\tld1w\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tst1w\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tldr\tw[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tstr\tw[0-9]+} } } */
+ /* Should multiply by (257-1)*4 rather than (VF-1)*4. */
+ /* { dg-final { scan-assembler-times {\tsbfiz\tx[0-9]+, x[0-9]+, 10, 32} 2 } } */
+ /* { dg-final { scan-assembler {\tcmp\tw2, 0} } } */
+ /* { dg-final { scan-assembler {\tcmp\tw3, 0} } } */
+ /* { dg-final { scan-assembler-times {\tcsel\tx[0-9]+} 4 } } */
+ /* Two range checks and a check for n being zero. (m being zero is OK.) */
+ /* { dg-final { scan-assembler {\tcmp\t} } } */
+ /* { dg-final { scan-assembler-times {\tccmp\t} 2 } } */
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_4_run.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_4_run.c 2017-11-17 22:11:53.267861313 +0000
***************
*** 0 ****
--- 1,18 ----
+ /* { dg-do run { target { aarch64_sve_hw } } } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #include "sve_var_stride_4.c"
+ #include "sve_var_stride_1.h"
+
+ int
+ main (void)
+ {
+ for (int n = -10; n < 10; ++n)
+ for (int m = -10; m < 10; ++m)
+ for (int offset = -17; offset <= 17; ++offset)
+ {
+ test (n, m, offset);
+ test (n, m, offset + n * (SIZE - 1));
+ }
+ return 0;
+ }
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_5.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_5.c 2017-11-17 22:11:53.267861313 +0000
***************
*** 0 ****
--- 1,27 ----
+ /* { dg-do compile } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #define TYPE double
+ #define SIZE 257
+
+ void __attribute__ ((weak))
+ f (TYPE *x, TYPE *y, long n, long m __attribute__((unused)))
+ {
+ for (int i = 0; i < SIZE; ++i)
+ x[i * n] += y[i * n];
+ }
+
+ /* { dg-final { scan-assembler {\tld1d\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tst1d\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tldr\td[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tstr\td[0-9]+} } } */
+ /* Should multiply by (VF-1)*8 rather than (257-1)*8. */
+ /* { dg-final { scan-assembler-not {, 2048} } } */
+ /* { dg-final { scan-assembler-not {\t.bfiz\t} } } */
+ /* { dg-final { scan-assembler-not {lsl[^\n]*[, ]11} } } */
+ /* { dg-final { scan-assembler {\tcmp\tx[0-9]+, 0} } } */
+ /* { dg-final { scan-assembler-not {\tcmp\tw[0-9]+, 0} } } */
+ /* { dg-final { scan-assembler-times {\tcsel\tx[0-9]+} 2 } } */
+ /* Two range checks and a check for n being zero. */
+ /* { dg-final { scan-assembler {\tcmp\t} } } */
+ /* { dg-final { scan-assembler-times {\tccmp\t} 2 } } */
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_5_run.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_5_run.c 2017-11-17 22:11:53.267861313 +0000
***************
*** 0 ****
--- 1,14 ----
+ /* { dg-do run { target { aarch64_sve_hw } } } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #include "sve_var_stride_5.c"
+ #include "sve_var_stride_1.h"
+
+ int
+ main (void)
+ {
+ for (int n = -10; n < 10; ++n)
+ for (int offset = -33; offset <= 33; ++offset)
+ test (n, n, offset);
+ return 0;
+ }
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_6.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_6.c 2017-11-17 22:11:53.267861313 +0000
***************
*** 0 ****
--- 1,25 ----
+ /* { dg-do compile } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #define TYPE long
+ #define SIZE 257
+
+ void __attribute__ ((weak))
+ f (TYPE *x, TYPE *y, long n, long m)
+ {
+ for (int i = 0; i < SIZE; ++i)
+ x[i * n] += y[i * m];
+ }
+
+ /* { dg-final { scan-assembler {\tld1d\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tst1d\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tldr\tx[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tstr\tx[0-9]+} } } */
+ /* Should multiply by (257-1)*8 rather than (VF-1)*8. */
+ /* { dg-final { scan-assembler-times {lsl\tx[0-9]+, x[0-9]+, 11} 2 } } */
+ /* { dg-final { scan-assembler {\tcmp\tx[0-9]+, 0} } } */
+ /* { dg-final { scan-assembler-not {\tcmp\tw[0-9]+, 0} } } */
+ /* { dg-final { scan-assembler-times {\tcsel\tx[0-9]+} 4 } } */
+ /* Two range checks and a check for n being zero. (m being zero is OK.) */
+ /* { dg-final { scan-assembler {\tcmp\t} } } */
+ /* { dg-final { scan-assembler-times {\tccmp\t} 2 } } */
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_6_run.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_6_run.c 2017-11-17 22:11:53.267861313 +0000
***************
*** 0 ****
--- 1,18 ----
+ /* { dg-do run { target { aarch64_sve_hw } } } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #include "sve_var_stride_6.c"
+ #include "sve_var_stride_1.h"
+
+ int
+ main (void)
+ {
+ for (int n = -10; n < 10; ++n)
+ for (int m = -10; m < 10; ++m)
+ for (int offset = -17; offset <= 17; ++offset)
+ {
+ test (n, m, offset);
+ test (n, m, offset + n * (SIZE - 1));
+ }
+ return 0;
+ }
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_7.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_7.c 2017-11-17 22:11:53.267861313 +0000
***************
*** 0 ****
--- 1,26 ----
+ /* { dg-do compile } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #define TYPE double
+ #define SIZE 257
+
+ void __attribute__ ((weak))
+ f (TYPE *x, TYPE *y, long n, long m __attribute__((unused)))
+ {
+ for (int i = 0; i < SIZE; ++i)
+ x[i * n] += y[i];
+ }
+
+ /* { dg-final { scan-assembler {\tld1d\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tst1d\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tldr\td[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tstr\td[0-9]+} } } */
+ /* Should multiply by (257-1)*8 rather than (VF-1)*8. */
+ /* { dg-final { scan-assembler-times {\tadd\tx[0-9]+, x1, 2048} 1 } } */
+ /* { dg-final { scan-assembler-times {lsl\tx[0-9]+, x[0-9]+, 11} 1 } } */
+ /* { dg-final { scan-assembler {\tcmp\tx[0-9]+, 0} } } */
+ /* { dg-final { scan-assembler-not {\tcmp\tw[0-9]+, 0} } } */
+ /* { dg-final { scan-assembler-times {\tcsel\tx[0-9]+} 2 } } */
+ /* Two range checks and a check for n being zero. */
+ /* { dg-final { scan-assembler {\tcmp\t} } } */
+ /* { dg-final { scan-assembler-times {\tccmp\t} 2 } } */
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_7_run.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_7_run.c 2017-11-17 22:11:53.267861313 +0000
***************
*** 0 ****
--- 1,14 ----
+ /* { dg-do run { target { aarch64_sve_hw } } } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #include "sve_var_stride_7.c"
+ #include "sve_var_stride_1.h"
+
+ int
+ main (void)
+ {
+ for (int n = -10; n < 10; ++n)
+ for (int offset = -33; offset <= 33; ++offset)
+ test (n, 1, offset);
+ return 0;
+ }
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_8.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_8.c 2017-11-17 22:11:53.267861313 +0000
***************
*** 0 ****
--- 1,26 ----
+ /* { dg-do compile } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #define TYPE long
+ #define SIZE 257
+
+ void
+ f (TYPE *x, TYPE *y, long n __attribute__((unused)), long m)
+ {
+ for (int i = 0; i < SIZE; ++i)
+ x[i] += y[i * m];
+ }
+
+ /* { dg-final { scan-assembler {\tld1d\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tst1d\tz[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tldr\tx[0-9]+} } } */
+ /* { dg-final { scan-assembler {\tstr\tx[0-9]+} } } */
+ /* Should multiply by (257-1)*8 rather than (VF-1)*8. */
+ /* { dg-final { scan-assembler-times {\tadd\tx[0-9]+, x0, 2048} 1 } } */
+ /* { dg-final { scan-assembler-times {lsl\tx[0-9]+, x[0-9]+, 11} 1 } } */
+ /* { dg-final { scan-assembler {\tcmp\tx[0-9]+, 0} } } */
+ /* { dg-final { scan-assembler-not {\tcmp\tw[0-9]+, 0} } } */
+ /* { dg-final { scan-assembler-times {\tcsel\tx[0-9]+} 2 } } */
+ /* Two range checks only; doesn't matter whether n is zero. */
+ /* { dg-final { scan-assembler {\tcmp\t} } } */
+ /* { dg-final { scan-assembler-times {\tccmp\t} 1 } } */
Index: gcc/testsuite/gcc.target/aarch64/sve_var_stride_8_run.c
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gcc.target/aarch64/sve_var_stride_8_run.c 2017-11-17 22:11:53.269720823 +0000
***************
*** 0 ****
--- 1,14 ----
+ /* { dg-do run { target { aarch64_sve_hw } } } */
+ /* { dg-options "-O2 -ftree-vectorize -march=armv8-a+sve" } */
+
+ #include "sve_var_stride_8.c"
+ #include "sve_var_stride_1.h"
+
+ int
+ main (void)
+ {
+ for (int n = -10; n < 10; ++n)
+ for (int offset = -33; offset <= 33; ++offset)
+ test (1, n, offset);
+ return 0;
+ }
Index: gcc/testsuite/gfortran.dg/vect/vect-alias-check-1.F90
===================================================================
*** /dev/null 2017-11-14 14:28:07.424493901 +0000
--- gcc/testsuite/gfortran.dg/vect/vect-alias-check-1.F90 2017-11-17 22:11:53.269720823 +0000
***************
*** 0 ****
--- 1,102 ----
+ ! { dg-do run }
+ ! { dg-additional-options "-fno-inline" }
+
+ #define N 200
+
+ #define TEST_VALUE(I) ((I) * 5 / 2)
+
+ subroutine setup(a)
+ real :: a(N)
+ do i = 1, N
+ a(i) = TEST_VALUE(i)
+ end do
+ end subroutine
+
+ subroutine check(a, x, gap)
+ real :: a(N), temp, x
+ integer :: gap
+ do i = 1, N - gap
+ temp = a(i + gap) + x
+ if (a(i) /= temp) call abort
+ end do
+ do i = N - gap + 1, N
+ temp = TEST_VALUE(i)
+ if (a(i) /= temp) call abort
+ end do
+ end subroutine
+
+ subroutine testa(a, x, base, n)
+ real :: a(n), x
+ integer :: base, n
+ do i = n, 2, -1
+ a(base + i - 1) = a(base + i) + x
+ end do
+ end subroutine testa
+
+ subroutine testb(a, x, base, n)
+ real :: a(n), x
+ integer :: base
+ do i = n, 4, -1
+ a(base + i - 3) = a(base + i) + x
+ end do
+ end subroutine testb
+
+ subroutine testc(a, x, base, n)
+ real :: a(n), x
+ integer :: base
+ do i = n, 8, -1
+ a(base + i - 7) = a(base + i) + x
+ end do
+ end subroutine testc
+
+ subroutine testd(a, x, base, n)
+ real :: a(n), x
+ integer :: base
+ do i = n, 16, -1
+ a(base + i - 15) = a(base + i) + x
+ end do
+ end subroutine testd
+
+ subroutine teste(a, x, base, n)
+ real :: a(n), x
+ integer :: base
+ do i = n, 32, -1
+ a(base + i - 31) = a(base + i) + x
+ end do
+ end subroutine teste
+
+ subroutine testf(a, x, base, n)
+ real :: a(n), x
+ integer :: base
+ do i = n, 64, -1
+ a(base + i - 63) = a(base + i) + x
+ end do
+ end subroutine testf
+
+ program main
+ real :: a(N)
+
+ call setup(a)
+ call testa(a, 91.0, 0, N)
+ call check(a, 91.0, 1)
+
+ call setup(a)
+ call testb(a, 55.0, 0, N)
+ call check(a, 55.0, 3)
+
+ call setup(a)
+ call testc(a, 72.0, 0, N)
+ call check(a, 72.0, 7)
+
+ call setup(a)
+ call testd(a, 69.0, 0, N)
+ call check(a, 69.0, 15)
+
+ call setup(a)
+ call teste(a, 44.0, 0, N)
+ call check(a, 44.0, 31)
+
+ call setup(a)
+ call testf(a, 39.0, 0, N)
+ call check(a, 39.0, 63)
+ end program
More information about the Gcc-patches
mailing list