[PATCH v2 2/4] postreload: forward loads across equal-size mode changes
ktkachov@nvidia.com
ktkachov@nvidia.com
Wed Oct 7 13:21:38 GMT 2026
From: Kyrylo Tkachov <ktkachov@nvidia.com>
cselib looks up memory values only in the mode of the access. A value
that is stored or loaded in one mode and then loaded in another mode of
the same size therefore misses a register that already holds it. For
example, SVE spill code can contain:
(set (mem:VNx4SI (plus:DI (reg:DI sp) (const_int K))) (reg:VNx4SI v7))
...
(set (reg:VNx2DI v23) (mem:VNx2DI (plus:DI (reg:DI sp) (const_int K))))
where the load can become:
(set (reg:VNx2DI v23) (reg:VNx2DI v7))
Scalar integer and floating-point modes have the same problem. DSE
forwards stored values across mode changes, but only before register
allocation, because it needs a new pseudo.
Add cselib_lookup_mem_other_modes, which returns the values recorded at
the address of a MEM in other modes of the same size. Split the filters
and the address lookup out of cselib_lookup_mem, so that both functions
use them. Like cselib_lookup_mem, ignore floating-point modes under
-ffloat-store.
In reload_cse_simplify_set, view the hard registers that hold these
values in the mode of the load with simplify_subreg, which checks
REG_CAN_CHANGE_MODE_P. Add them to the existing candidates, so that the
existing cost comparison, LOAD_EXTEND_OP handling and instruction
validation apply. As in LRA and regcprop, this relies on
REG_CAN_CHANGE_MODE_P to allow a mode change only if the register then
holds the bytes that memory would hold in the new mode.
The motivating code is a dav1d loop filter compiled
with -march=armv9-a+sve2 -msve-vector-bits=256.
A testcase derived from it is given in
gcc.target/aarch64/sve/spill-mode-change.c.
At -O3 this patch removes 12% of its SVE vector loads (ldr z), and 8% with -fno-ivopts.
All the removed loads are reloads of spill slots. The instruction count
of the aarch64 cc1 for -O2 compiles of combine.ii, dwarf2out.ii,
fold-const.ii and sqlite3.i changes by -0.01% to +0.01%.
Bootstrapped and tested on aarch64-none-linux-gnu and x86_64-linux.
Ok for trunk?
Thanks,
Kyrill
Cc: Jeff Law <jeffrey.law@oss.qualcomm.com>
gcc/ChangeLog:
* cselib.cc (cselib_lookup_mem_addr): New function, split out of...
(cselib_lookup_mem): ...here.
(cselib_lookup_mem_other_modes): New function.
* cselib.h (cselib_lookup_mem_other_modes): Declare.
* postreload.cc (reload_cse_add_mem_regs): New function.
(reload_cse_simplify_set): Also consider registers that hold the
loaded value in another mode of the same size.
gcc/testsuite/ChangeLog:
* gcc.dg/rtl/aarch64/postreload-mode-forward.c: New test.
* gcc.dg/rtl/aarch64/postreload-mode-forward-be.c: Likewise.
* gcc.dg/rtl/aarch64/postreload-mode-forward-call.c: Likewise.
* gcc.dg/rtl/aarch64/postreload-mode-forward-extend.c: Likewise.
* gcc.dg/rtl/aarch64/postreload-mode-forward-float-store.c: Likewise.
* gcc.target/aarch64/sve/spill-mode-change.c: Likewise.
* gcc.target/i386/addr-space-10.c: Likewise.
Signed-off-by: Kyrylo Tkachov <ktkachov@nvidia.com>
---
Changes since v1:
- Handle all modes, not only vector modes (Jeff).
- Add a cselib query instead of exporting elt_list and copying the
address lookup of cselib_lookup_mem.
- Add the candidates to the existing loop in reload_cse_simplify_set,
so that the cost check and the LOAD_EXTEND_OP handling apply.
- Apply -ffloat-store to the stored mode too.
- More tests: scalar modes, calls, -ffloat-store, big-endian, a mode
change that the target rejects, x86 address spaces.
gcc/cselib.cc | 64 ++++-
gcc/cselib.h | 1 +
gcc/postreload.cc | 34 ++-
.../rtl/aarch64/postreload-mode-forward-be.c | 29 +++
.../aarch64/postreload-mode-forward-call.c | 70 ++++++
.../aarch64/postreload-mode-forward-extend.c | 62 +++++
.../postreload-mode-forward-float-store.c | 102 ++++++++
.../rtl/aarch64/postreload-mode-forward.c | 228 ++++++++++++++++++
.../aarch64/sve/spill-mode-change.c | 95 ++++++++
gcc/testsuite/gcc.target/i386/addr-space-10.c | 39 +++
10 files changed, 708 insertions(+), 16 deletions(-)
create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-be.c
create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-call.c
create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-extend.c
create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-float-store.c
create mode 100644 gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward.c
create mode 100644 gcc/testsuite/gcc.target/aarch64/sve/spill-mode-change.c
create mode 100644 gcc/testsuite/gcc.target/i386/addr-space-10.c
diff --git a/gcc/cselib.cc b/gcc/cselib.cc
index 5567caa47..020fecb2d 100644
--- a/gcc/cselib.cc
+++ b/gcc/cselib.cc
@@ -1765,6 +1765,30 @@ add_mem_for_addr (cselib_val *addr_elt, cselib_val *mem_elt, rtx x)
}
}
+/* Return the canonical value for the address of X, which is a MEM rtx.
+ If CREATE, make a new value for the address if we haven't seen it
+ before. Return null if cselib does not record X or if the address
+ has no value. */
+
+static cselib_val *
+cselib_lookup_mem_addr (rtx x, int create)
+{
+ machine_mode mode = GET_MODE (x);
+
+ if (MEM_VOLATILE_P (x) || mode == BLKmode
+ || !cselib_record_memory
+ || (FLOAT_MODE_P (mode) && flag_float_store))
+ return 0;
+
+ machine_mode addr_mode = GET_MODE (XEXP (x, 0));
+ if (addr_mode == VOIDmode)
+ addr_mode = Pmode;
+
+ /* Look up the value for the address. */
+ cselib_val *addr = cselib_lookup (XEXP (x, 0), addr_mode, create, mode);
+ return addr ? canonical_cselib_val (addr) : 0;
+}
+
/* Return true if V has a memory location at address ADDR in address
space AS. */
@@ -1790,25 +1814,13 @@ static cselib_val *
cselib_lookup_mem (rtx x, int create)
{
machine_mode mode = GET_MODE (x);
- machine_mode addr_mode;
cselib_val **slot;
cselib_val *addr;
cselib_val *mem_elt;
- if (MEM_VOLATILE_P (x) || mode == BLKmode
- || !cselib_record_memory
- || (FLOAT_MODE_P (mode) && flag_float_store))
- return 0;
-
- addr_mode = GET_MODE (XEXP (x, 0));
- if (addr_mode == VOIDmode)
- addr_mode = Pmode;
-
- /* Look up the value for the address. */
- addr = cselib_lookup (XEXP (x, 0), addr_mode, create, mode);
+ addr = cselib_lookup_mem_addr (x, create);
if (! addr)
return 0;
- addr = canonical_cselib_val (addr);
/* Find a value that describes a value of our mode at that address. */
for (elt_list *l = addr->addr_list; l; l = l->next)
@@ -1829,6 +1841,32 @@ cselib_lookup_mem (rtx x, int create)
return mem_elt;
}
+/* Add to VALUES the values that describe the contents of X, which is a MEM
+ rtx, in other modes of the same size. Like cselib_lookup_mem, ignore
+ floating-point modes if -ffloat-store. */
+
+void
+cselib_lookup_mem_other_modes (rtx x, vec<cselib_val *> &values)
+{
+ cselib_val *addr = cselib_lookup_mem_addr (x, 0);
+ if (!addr)
+ return;
+
+ machine_mode mode = GET_MODE (x);
+ for (elt_list *l = addr->addr_list; l; l = l->next)
+ {
+ machine_mode val_mode = GET_MODE (l->elt->val_rtx);
+ if (val_mode != mode
+ && known_eq (GET_MODE_SIZE (val_mode), GET_MODE_SIZE (mode))
+ && !(FLOAT_MODE_P (val_mode) && flag_float_store)
+ && cselib_has_mem_loc_p (l->elt, addr, MEM_ADDR_SPACE (x)))
+ {
+ promote_debug_loc (l->elt->locs);
+ values.safe_push (l->elt);
+ }
+ }
+}
+
/* Search through the possible substitutions in P. We prefer a non reg
substitution because this allows us to expand the tree further. If
we find, just a reg, take the lowest regno. There may be several
diff --git a/gcc/cselib.h b/gcc/cselib.h
index 6493303f7..156b57d79 100644
--- a/gcc/cselib.h
+++ b/gcc/cselib.h
@@ -80,6 +80,7 @@ extern cselib_val *cselib_lookup (rtx, machine_mode,
int, machine_mode);
extern cselib_val *cselib_lookup_from_insn (rtx, machine_mode,
int, machine_mode, rtx_insn *);
+extern void cselib_lookup_mem_other_modes (rtx, vec<cselib_val *> &);
extern void cselib_init (int);
extern void cselib_clear_table (void);
extern void cselib_finish (void);
diff --git a/gcc/postreload.cc b/gcc/postreload.cc
index 475b58896..0e837d91b 100644
--- a/gcc/postreload.cc
+++ b/gcc/postreload.cc
@@ -355,6 +355,29 @@ reload_cse_regs_1 (void)
cleanup_cfg (0);
}
+/* Add to LOCS the hard registers that hold the contents of MEM, a memory
+ reference, in other modes of the same size. Express each register
+ in the mode of MEM. */
+
+static void
+reload_cse_add_mem_regs (rtx mem, vec<rtx> &locs)
+{
+ auto_vec<cselib_val *, 4> values;
+ cselib_lookup_mem_other_modes (mem, values);
+ for (cselib_val *v : values)
+ for (elt_loc_list *l = v->locs; l; l = l->next)
+ if (REG_P (l->loc)
+ /* The target must allow the bitcast in the register. */
+ && REG_CAN_CHANGE_MODE_P (REGNO (l->loc), GET_MODE (l->loc),
+ GET_MODE (mem)))
+ {
+ rtx reg = simplify_subreg (GET_MODE (mem), l->loc,
+ GET_MODE (l->loc), 0);
+ if (reg && REG_P (reg))
+ locs.safe_push (reg);
+ }
+}
+
/* Try to simplify a single SET instruction. SET is the set pattern.
INSN is the instruction it came from.
This function only handles one case: if we set a register to a value
@@ -394,7 +417,13 @@ reload_cse_simplify_set (rtx set, rtx_insn *insn)
return 0;
val = cselib_lookup (src, GET_MODE (SET_DEST (set)), 0, VOIDmode);
- if (! val)
+ auto_vec<rtx, 8> locs;
+ if (val)
+ for (l = val->locs; l; l = l->next)
+ locs.safe_push (l->loc);
+ if (MEM_P (src))
+ reload_cse_add_mem_regs (src, locs);
+ if (locs.is_empty ())
return 0;
/* If memory loads are cheaper than register copies, don't change them. */
@@ -406,9 +435,8 @@ reload_cse_simplify_set (rtx set, rtx_insn *insn)
else
old_cost = set_src_cost (src, GET_MODE (SET_DEST (set)), speed);
- for (l = val->locs; l; l = l->next)
+ for (rtx this_rtx : locs)
{
- rtx this_rtx = l->loc;
int this_cost;
if (CONSTANT_P (this_rtx) && ! references_value_p (this_rtx))
diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-be.c b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-be.c
new file mode 100644
index 000000000..d7a65d51a
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-be.c
@@ -0,0 +1,29 @@
+/* { dg-do compile { target aarch64*-*-* } } */
+/* { dg-options "-O2 -mbig-endian" } */
+/* { dg-final { check-function-bodies "**" "" "" } } */
+
+/* On big-endian targets, Advanced SIMD registers still allow a change
+ between vector modes of the same size. */
+
+/*
+** forward_advsimd:
+** str q0, \[x0\]
+** str q0, \[x1\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+forward_advsimd (void)
+{
+(function "forward_advsimd"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:V4SI (reg:DI x0) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 3 (set (reg:V2DI v1) (mem:V2DI (reg:DI x0) [0 S16 A128])))
+ (cinsn 4 (set (mem:V2DI (reg:DI x1) [0 S16 A128]) (reg:V2DI v1)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-call.c b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-call.c
new file mode 100644
index 000000000..585479df4
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-call.c
@@ -0,0 +1,70 @@
+/* { dg-do compile { target aarch64*-*-* } } */
+/* { dg-options "-O2" } */
+/* { dg-final { check-function-bodies "**" "" "" } } */
+
+/* A call to a const function preserves the low 64 bits of V8 but not
+ the full 128-bit register. */
+
+/*
+** preserved:
+** ...
+** str d8, \[x19\]
+** bl f
+** str d8, \[x20\]
+** ...
+*/
+void __RTL (startwith ("ira"))
+preserved (void)
+{
+(function "preserved"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:V8QI (reg:DI x19) [0 S8 A64]) (reg:V8QI v8)))
+ (ccall_insn/u 3 (parallel
+ [(set (reg:DI x0)
+ (call (mem:DI (symbol_ref:DI ("f")) [0 S4 A32])
+ (const_int 0)))
+ (unspec:DI [(const_int 2)] UNSPEC_CALLEE_ABI)
+ (clobber (reg:DI x30))]) (nil) (nil) 0)
+ (cinsn 4 (set (reg:V4HI v1) (mem:V4HI (reg:DI x19) [0 S8 A64])))
+ (cinsn 5 (set (mem:V4HI (reg:DI x20) [0 S8 A64]) (reg:V4HI v1)))
+ (cinsn 6 (set (mem:DI (reg:DI x21) [0 S8 A64]) (reg:DI x0)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/*
+** clobbered:
+** ...
+** str q8, \[x19\]
+** bl f
+** ldr q[0-9]+, \[x19\]
+** ...
+*/
+void __RTL (startwith ("ira"))
+clobbered (void)
+{
+(function "clobbered"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:V16QI (reg:DI x19) [0 S16 A128]) (reg:V16QI v8)))
+ (ccall_insn/u 3 (parallel
+ [(set (reg:DI x0)
+ (call (mem:DI (symbol_ref:DI ("f")) [0 S4 A32])
+ (const_int 0)))
+ (unspec:DI [(const_int 2)] UNSPEC_CALLEE_ABI)
+ (clobber (reg:DI x30))]) (nil) (nil) 0)
+ (cinsn 4 (set (reg:V8HI v1) (mem:V8HI (reg:DI x19) [0 S16 A128])))
+ (cinsn 5 (set (mem:V8HI (reg:DI x20) [0 S16 A128]) (reg:V8HI v1)))
+ (cinsn 6 (set (mem:DI (reg:DI x21) [0 S8 A64]) (reg:DI x0)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-extend.c b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-extend.c
new file mode 100644
index 000000000..c630c74a1
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-extend.c
@@ -0,0 +1,62 @@
+/* { dg-do compile { target aarch64*-*-* } } */
+/* { dg-options "-O2 -mtune=xgene1 -fdump-rtl-postreload" } */
+
+/* AArch64 loads of narrow integers zero-extend to 64 bits (LOAD_EXTEND_OP).
+ A register that replaces such a load must keep the extension. xgene1
+ makes loads more expensive than extensions. */
+
+void __RTL (startwith ("ira"))
+sf_to_si (void)
+{
+(function "sf_to_si"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:SF (reg:DI x0) [0 S4 A32]) (reg:SF v0)))
+ (cinsn 3 (set (reg:SI x3) (mem:SI (reg:DI x0) [0 S4 A32])))
+ (cinsn 4 (set (mem:DI (reg:DI x1) [0 S8 A64]) (reg:DI x3)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+void __RTL (startwith ("ira"))
+hf_to_hi (void)
+{
+(function "hf_to_hi"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:HF (reg:DI x0) [0 S2 A16]) (reg:HF v1)))
+ (cinsn 3 (set (reg:HI x3) (mem:HI (reg:DI x0) [0 S2 A16])))
+ (cinsn 4 (set (mem:DI (reg:DI x1) [0 S8 A64]) (reg:DI x3)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+void __RTL (startwith ("ira"))
+gpr_sf_to_si (void)
+{
+(function "gpr_sf_to_si"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:SF (reg:DI x0) [0 S4 A32]) (reg:SF x2)))
+ (cinsn 3 (set (reg:SI x3) (mem:SI (reg:DI x0) [0 S4 A32])))
+ (cinsn 4 (set (mem:DI (reg:DI x1) [0 S8 A64]) (reg:DI x3)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* { dg-final { scan-rtl-dump {\(set \(reg:DI 3 x3\)\n\s+\(zero_extend:DI \(reg:SI 32 v0\)\)\)} "postreload" } } */
+/* { dg-final { scan-rtl-dump {\(set \(reg:DI 3 x3\)\n\s+\(zero_extend:DI \(reg:HI 33 v1\)\)\)} "postreload" } } */
+/* { dg-final { scan-rtl-dump {\(set \(reg:DI 3 x3\)\n\s+\(zero_extend:DI \(reg:SI 2 x2\)\)\)} "postreload" } } */
+/* { dg-final { scan-assembler-not {\tldrh?\tw3} } } */
diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-float-store.c b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-float-store.c
new file mode 100644
index 000000000..f57cd523f
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward-float-store.c
@@ -0,0 +1,102 @@
+/* { dg-do compile { target aarch64*-*-* } } */
+/* { dg-options "-O2 -ffloat-store" } */
+/* { dg-final { check-function-bodies "**" "" "" } } */
+
+/* -ffloat-store prevents forwarding into and out of floating-point modes,
+ including vector ones. */
+
+/*
+** int_to_float:
+** str x2, \[x0\]
+** ldr x3, \[x0\]
+** str x3, \[x1\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+int_to_float (void)
+{
+(function "int_to_float"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:DI (reg:DI x0) [0 S8 A64]) (reg:DI x2)))
+ (cinsn 3 (set (reg:DF x3) (mem:DF (reg:DI x0) [0 S8 A64])))
+ (cinsn 4 (set (mem:DF (reg:DI x1) [0 S8 A64]) (reg:DF x3)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/*
+** float_to_int:
+** str x2, \[x0\]
+** ldr x3, \[x0\]
+** str x3, \[x1\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+float_to_int (void)
+{
+(function "float_to_int"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:DF (reg:DI x0) [0 S8 A64]) (reg:DF x2)))
+ (cinsn 3 (set (reg:DI x3) (mem:DI (reg:DI x0) [0 S8 A64])))
+ (cinsn 4 (set (mem:DI (reg:DI x1) [0 S8 A64]) (reg:DI x3)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/*
+** int_to_float_vector:
+** str q0, \[x0\]
+** ldr q1, \[x0\]
+** str q1, \[x1\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+int_to_float_vector (void)
+{
+(function "int_to_float_vector"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:V4SI (reg:DI x0) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 3 (set (reg:V4SF v1) (mem:V4SF (reg:DI x0) [0 S16 A128])))
+ (cinsn 4 (set (mem:V4SF (reg:DI x1) [0 S16 A128]) (reg:V4SF v1)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/*
+** float_to_int_vector:
+** str q0, \[x0\]
+** ldr q1, \[x0\]
+** str q1, \[x1\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+float_to_int_vector (void)
+{
+(function "float_to_int_vector"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:V4SF (reg:DI x0) [0 S16 A128]) (reg:V4SF v0)))
+ (cinsn 3 (set (reg:V4SI v1) (mem:V4SI (reg:DI x0) [0 S16 A128])))
+ (cinsn 4 (set (mem:V4SI (reg:DI x1) [0 S16 A128]) (reg:V4SI v1)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
diff --git a/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward.c b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward.c
new file mode 100644
index 000000000..93db5e4c9
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/rtl/aarch64/postreload-mode-forward.c
@@ -0,0 +1,228 @@
+/* { dg-do compile { target aarch64*-*-* } } */
+/* { dg-options "-O2 -mtune=generic-armv9-a -fno-schedule-insns2" } */
+/* { dg-final { check-function-bodies "**" "" "" } } */
+
+/* Postreload replaces a load with a register that holds the loaded bytes
+ in another mode of the same size. generic-armv9-a makes moves between
+ general and FP registers cheaper than loads. */
+
+/*
+** int_to_float:
+** str x2, \[x0\]
+** str x2, \[x1\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+int_to_float (void)
+{
+(function "int_to_float"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:DI (reg:DI x0) [0 S8 A64]) (reg:DI x2)))
+ (cinsn 3 (set (reg:DF v0) (mem:DF (reg:DI x0) [0 S8 A64])))
+ (cinsn 4 (set (mem:DF (reg:DI x1) [0 S8 A64]) (reg:DF v0)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/*
+** float_to_int:
+** str x2, \[x0\]
+** str x2, \[x1\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+float_to_int (void)
+{
+(function "float_to_int"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:DF (reg:DI x0) [0 S8 A64]) (reg:DF x2)))
+ (cinsn 3 (set (reg:DI x3) (mem:DI (reg:DI x0) [0 S8 A64])))
+ (cinsn 4 (set (mem:DI (reg:DI x1) [0 S8 A64]) (reg:DI x3)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/*
+** vector_elements:
+** str q0, \[x0\]
+** str q0, \[x1\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+vector_elements (void)
+{
+(function "vector_elements"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:V4SI (reg:DI x0) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 3 (set (reg:V2DI v1) (mem:V2DI (reg:DI x0) [0 S16 A128])))
+ (cinsn 4 (set (mem:V2DI (reg:DI x1) [0 S16 A128]) (reg:V2DI v1)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/*
+** scalar_to_vector:
+** str q0, \[x0\]
+** str q0, \[x1\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+scalar_to_vector (void)
+{
+(function "scalar_to_vector"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:TI (reg:DI x0) [0 S16 A128]) (reg:TI v0)))
+ (cinsn 3 (set (reg:V2DI v1) (mem:V2DI (reg:DI x0) [0 S16 A128])))
+ (cinsn 4 (set (mem:V2DI (reg:DI x1) [0 S16 A128]) (reg:V2DI v1)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/*
+** gpr_pair:
+** stp x2, x3, \[x0\]
+** mov x4, x2
+** mov x5, x3
+** stp x4, x5, \[x1\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+gpr_pair (void)
+{
+(function "gpr_pair"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:TI (reg:DI x0) [0 S16 A128]) (reg:TI x2)))
+ (cinsn 3 (set (reg:TF x4) (mem:TF (reg:DI x0) [0 S16 A128])))
+ (cinsn 4 (set (mem:TF (reg:DI x1) [0 S16 A128]) (reg:TF x4)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* The load also takes the value from an earlier load in another mode. */
+
+/*
+** load_to_load:
+** ldr q0, \[x0\]
+** str q0, \[x1\]
+** str q0, \[x2\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+load_to_load (void)
+{
+(function "load_to_load"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (reg:TI v0) (mem:TI (reg:DI x0) [0 S16 A128])))
+ (cinsn 3 (set (reg:V2DI v1) (mem:V2DI (reg:DI x0) [0 S16 A128])))
+ (cinsn 4 (set (mem:TI (reg:DI x1) [0 S16 A128]) (reg:TI v0)))
+ (cinsn 5 (set (mem:V2DI (reg:DI x2) [0 S16 A128]) (reg:V2DI v1)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/*
+** clobbered:
+** str q0, \[x0\]
+** ldr q1, \[x0\]
+** str q1, \[x1\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+clobbered (void)
+{
+(function "clobbered"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:V4SI (reg:DI x0) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 3 (set (reg:V4SI v0) (reg:V4SI v2)))
+ (cinsn 4 (set (reg:V2DI v1) (mem:V2DI (reg:DI x0) [0 S16 A128])))
+ (cinsn 5 (set (mem:V2DI (reg:DI x1) [0 S16 A128]) (reg:V2DI v1)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/*
+** alias_store:
+** str q0, \[x0\]
+** str w2, \[x0\]
+** ldr q1, \[x0\]
+** str q1, \[x1\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+alias_store (void)
+{
+(function "alias_store"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:V4SI (reg:DI x0) [0 S16 A128]) (reg:V4SI v0)))
+ (cinsn 3 (set (mem:SI (reg:DI x0) [0 S4 A32]) (reg:SI x2)))
+ (cinsn 4 (set (reg:V2DI v1) (mem:V2DI (reg:DI x0) [0 S16 A128])))
+ (cinsn 5 (set (mem:V2DI (reg:DI x1) [0 S16 A128]) (reg:V2DI v1)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
+
+/* V2x8QI occupies two D registers, so Q0 does not hold the V16QI value. */
+
+/*
+** partial_struct:
+** stp d0, d1, \[x0\]
+** ldr q2, \[x0\]
+** str q2, \[x1\]
+** ret
+*/
+void __RTL (startwith ("ira"))
+partial_struct (void)
+{
+(function "partial_struct"
+ (insn-chain
+ (block 2
+ (edge-from entry (flags "FALLTHRU"))
+ (cnote 1 [bb 2] NOTE_INSN_BASIC_BLOCK)
+ (cinsn 2 (set (mem:V2x8QI (reg:DI x0) [0 S16 A64]) (reg:V2x8QI v0)))
+ (cinsn 3 (set (reg:V16QI v2) (mem:V16QI (reg:DI x0) [0 S16 A128])))
+ (cinsn 4 (set (mem:V16QI (reg:DI x1) [0 S16 A128]) (reg:V16QI v2)))
+ (edge-to exit (flags "FALLTHRU"))
+ )
+ )
+)
+}
diff --git a/gcc/testsuite/gcc.target/aarch64/sve/spill-mode-change.c b/gcc/testsuite/gcc.target/aarch64/sve/spill-mode-change.c
new file mode 100644
index 000000000..0d10c047d
--- /dev/null
+++ b/gcc/testsuite/gcc.target/aarch64/sve/spill-mode-change.c
@@ -0,0 +1,95 @@
+/* { dg-do compile } */
+/* { dg-options "-O3 -march=armv9-a -mtune=generic-armv9-a -msve-vector-bits=256 -fno-ivopts" } */
+/* { dg-require-effective-target aarch64_little_endian } */
+
+/* Reuse a stored vector across a change in element size. */
+
+#include <arm_sve.h>
+
+typedef svuint8_t v8 __attribute__ ((arm_sve_vector_bits (256)));
+
+static inline void
+f (v8 *a)
+{
+ for (int i = 0; i < 16; i += 8)
+ for (int j = 0; j < 4; j++)
+ {
+ svuint32_t x = svtrn1_u32 (svreinterpret_u32 (a[i + j]),
+ svreinterpret_u32 (a[i + j + 4]));
+ svuint32_t y = svtrn2_u32 (svreinterpret_u32 (a[j]),
+ svreinterpret_u32 (a[i + j]));
+ a[i + j] = svreinterpret_u8 (x);
+ a[i + j + 4] = svreinterpret_u8 (y);
+ }
+ for (int i = 0; i < 8; i++)
+ {
+ v8 x = svreinterpret_u8 (svtrn1_u64 (svreinterpret_u64 (a[i]),
+ svreinterpret_u64 (a[i + 8])));
+ a[i + 8] = svreinterpret_u8 (svtrn2_u64 (svreinterpret_u64 (a[i]),
+ svreinterpret_u64 (a[i + 8])));
+ a[i] = x;
+ }
+}
+
+void
+g (uint8_t *a, long b, uint32_t *c, uint64_t *d, uint8_t *e)
+{
+ unsigned h = c[10];
+ svbool_t p = svptrue_pat_b8 (SV_VL32);
+ svbool_t q = svptrue_pat_b8 (SV_VL16);
+ svbool_t r = svptrue_pat_b32 (SV_VL8);
+ v8 t = svld1_u8 (p, e);
+#define T(x) svtbl_u8 (svuzp1_u8 (svuzp1_u8 (svreinterpret_u8 (x), \
+ svdup_u8 (0)), svdup_u8 (0)), t)
+ uint32_t k = d[1];
+ for (int i = 0; i < 32; i += 8)
+ {
+ v8 v0 = svreinterpret_u8 (svinsr_n_u64 (svdup_u64 (0), 0));
+ svuint32_t v1 = svunpklo_u32 (svunpklo_u16 (v0));
+ svuint32_t v2 = svadd_n_u32_x (r, svindex_u32 (0, 1), i);
+ svuint32_t v3 = svlsl_u32_x (r, svdup_u32 (1), v2);
+ svbool_t p0 = svand_z (
+ r, svcmpne_n_u32 (r, svand_u32_x (r, svdup_u32 (h), v3), 0),
+ svcmpne_n_u32 (r, v1, 0));
+ svuint32_t v4 = svsel_u32 (p0, svdup_u32 (1), svdup_u32 (0));
+ svbool_t p1 = svcmpne_n_u8 (p, T (v4), 0);
+ if (svptest_any (p, p1))
+ continue;
+ uint8_t *s = a + b, *u = s + 7;
+ v8 v[16];
+ for (int j = 0; j < 16; j++)
+ {
+ v[j] = svsplice_u8 (q, svld1_u8 (q, s), svld1_u8 (q, u));
+ u += b;
+ }
+ f (v);
+ svuint32_t v5 = svmax_n_u32_x (r, svdup_u32 (k), 1);
+ svuint32_t v6 = svlsr_n_u32_x (r, v1, 4);
+ svbool_t p2 = svcmpne_n_u32 (
+ r, svand_u32_x (r, svdup_u32 (c[1]), v3), 0);
+ svuint32_t v7 = svsel_u32 (p2, svdup_u32 (8), svdup_u32 (4));
+ v8 v15 = T (v5), v9 = T (v6), v10 = T (v7);
+ svbool_t p3 = svcmple_u8 (p, v[7], v15);
+ v8 v11 = svabd_u8_x (p, v[6], v[7]);
+ v11 = svqadd_u8 (v11, svlsr_n_u8_x (
+ p, svabd_u8_x (p, v[9], v[8]), 1));
+ svbool_t p4 = svcmple_u8 (p, v11, v15);
+ svbool_t p5 = svand_z (p, svand_z (p, p3, p4), p1);
+ if (svptest_any (p, p5))
+ continue;
+ svbool_t p6 = svcmpgt_u8 (p, v[7], v9);
+ svbool_t p7 = svcmpge_n_u8 (p, v10, 6);
+ v8 v12 = svdup_u8 (128);
+ svint8_t v13 = svqsub_s8_m (p6, svreinterpret_s8 (v12), svdup_s8 (0));
+ v8 v14 = sveor_u8_x (p, svreinterpret_u8 (v13), v12);
+ v[8] = svsel_u8 (p7, v14, v[8]);
+ for (int j = 0; j < 16; j++)
+ {
+ svst1_u8 (q, s, v[j]);
+ s += b;
+ }
+ }
+}
+
+/* { dg-final { scan-assembler-not {\tstr\tz[0-9]+, (\[sp, #[0-9]+, mul vl\])\n\tldr\tz[0-9]+, \1} } } */
+/* { dg-final { scan-assembler {\tstr\tz[0-9]+, \[sp, #[0-9]+, mul vl\]} } } */
diff --git a/gcc/testsuite/gcc.target/i386/addr-space-10.c b/gcc/testsuite/gcc.target/i386/addr-space-10.c
new file mode 100644
index 000000000..229a4bb03
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/addr-space-10.c
@@ -0,0 +1,39 @@
+/* { dg-do compile { target { ! ia32 } } } */
+/* { dg-options "-O2 -fno-dse" } */
+
+/* *P and *Q are different memory locations with the same address. The
+ stored values are also in generic memory locations. A load from Q in
+ another mode of the same size must not reuse those values. */
+
+typedef int v4si __attribute__ ((vector_size (16)));
+typedef float v4sf __attribute__ ((vector_size (16)));
+
+static const long long tabl[4] = { 1, 2, 3, 4 };
+static const int tabi[4] = { 1, 2, 3, 4 };
+static const v4si tabv[2] = { { 1, 2, 3, 4 }, { 5, 6, 7, 8 } };
+
+double
+f (int i, long long __seg_gs *p)
+{
+ double *q = (double *) (__UINTPTR_TYPE__) p;
+ *p = tabl[i];
+ return *q;
+}
+
+float
+g (int i, int __seg_gs *p)
+{
+ float *q = (float *) (__UINTPTR_TYPE__) p;
+ *p = tabi[i];
+ return *q;
+}
+
+v4sf
+h (int i, v4si __seg_gs *p)
+{
+ v4sf *q = (v4sf *) (__UINTPTR_TYPE__) p;
+ *p = tabv[i];
+ return *q;
+}
+
+/* { dg-final { scan-assembler-times {\t[a-z]+\t\(%[a-z0-9]+\), %} 3 } } */
--
2.43.0
More information about the Gcc-patches
mailing list