[gcc r17-3709] gimple-fold: Fix up len_load/len_store folding [PR127100]
Jakub Jelinek
jakub@gcc.gnu.org
Thu Aug 27 18:33:09 GMT 2026
https://gcc.gnu.org/g:1dc229c7368d4f8b43356968f624e0d895961868
commit r17-3709-g1dc229c7368d4f8b43356968f624e0d895961868
Author: Jakub Jelinek <jakub@redhat.com>
Date: Thu Aug 27 20:22:55 2026 +0200
gimple-fold: Fix up len_load/len_store folding [PR127100]
There are various optabs and corresponding internal functions which
take 2 separate arguments, length (counted in number of vector elements)
and bias. The bias is documented to be only 0 or -1 and is a property
of the target. Out of rs6000, s390 and riscv backends which use any of
these optabs/ifns, only s390 has bias -1 (and only defines len_load/len_store
optabs), the other targets have bias 0.
Now, before r14-1932 it has been documented that the optabs load/store
len - bias elements, on most targets that just means len elements, except
on s390 z13+ it actually loads the provided operand + 1 elements.
There is one spot in sccvn which correctly uses
pd.size = (tree_to_uhwi (len)
+ -tree_to_shwi (bias)) * BITS_PER_UNIT;
r14-1932 changed the documentation to use len + bias instead for unclear
reassons, but otherwise didn't change the bias handling at all.
And then r16-5984 probably followed the adjusted documentation and uses
wlen = wi::to_poly_widest (len) + wi::to_widest (bias);
That is wrong. On the intrinsic_intkinds_1.f90 testcase, we have several
.LEN_LOAD calls which have 1, -1 as the last two arguments and it actually
means it loads 2 bytes, but partial_load_store_mask_state instead computes
1 + -1 = 0 and assumes it is all inactive and folds it away.
The following patch adjusts the documentation in all places where bias is
used I could find (admittedly, most of the optabs are only defined on riscv
right now and the bias is 0 there, so it doesn't make a difference), fixes some
pastos in operand numbers and finally fixes up the r16-5984 change to
compute len - bias instead.
2026-08-27 Jakub Jelinek <jakub@redhat.com>
PR tree-optimization/127100
* doc/md.texi (vec_mask_len_load_lanes@var{m}@var{n}): Use
operand4 - operand5 rather than operand4 + operand5.
(vec_mask_len_store_lanes@var{m}@var{n}): Use
operand3 - operand4 rather than operand3 + operand4.
(mask_len_gather_load@var{m}@var{n}): Use operand 7 - operand 8
rather than operand 7 + operand 8.
(mask_len_strided_load@var{m}): Use operand 4 - operand 5 rather
than operand 4 + operand 5.
(mask_len_scatter_store@var{m}@var{n}): Use operand 6 - operand 7
rather than operand 6 + operand 7.
(mask_len_strided_store@var{m}): Use operand 4 - operand 5 rather
than operand 4 + operand 5.
(len_load_@var{m}): Use operand 3 - operand 4 rather than
operand 3 + operand 4.
(len_store_@var{m}): Use operand 2 - operand 3 rather than
operand 2 + operand 3.
(mask_len_load@var{m}@var{n}): Use operand 4 - operand 5 rather
than operand 3 + operand 4 or operand 4 + operand 5. Use
Operand 5 rather than Operand 4 when talking about bias QI mode.
(mask_len_store@var{m}@var{n}): Use operand 3 - operand 4 rather
than operand 3 + operand 4 or operand 2 + operand 4.
(cond_len_neg@var{mode}): Use operand 4 - operand 5 rather than
operand 4 + operand 5. Use ops[4] - ops[5] rather than
ops[4] + ops[5].
(cond_len_add@var{mode}): Use operand 5 - operand 6 rather than
operand 5 + operand 6. Use ops[5] - ops[6] rather than
ops[5] + ops[6].
(cond_len_fma@var{mode}): Use ops[6] - ops[7] rather than
ops[6] + ops[7].
* gimple-fold.cc (partial_load_store_mask_state): Compute wlen
as wi::to_poly_widest (len) - wi::to_widest (bias) rather than
wi::to_poly_widest (len) + wi::to_widest (bias).
* gfortran.dg/pr127100.f90: New test.
Reviewed-by: Richard Biener <rguenth@suse.de>
Diff:
---
gcc/doc/md.texi | 52 +++++++++++++++++-----------------
gcc/gimple-fold.cc | 2 +-
gcc/testsuite/gfortran.dg/pr127100.f90 | 11 +++++++
3 files changed, 38 insertions(+), 27 deletions(-)
diff --git a/gcc/doc/md.texi b/gcc/doc/md.texi
index f86fa8038e8a..80761b0a8a62 100644
--- a/gcc/doc/md.texi
+++ b/gcc/doc/md.texi
@@ -7341,7 +7341,7 @@ The operation is equivalent to:
@smallexample
int c = GET_MODE_SIZE (@var{m}) / GET_MODE_SIZE (@var{n});
-for (j = 0; j < operand4 + operand5; j++)
+for (j = 0; j < operand4 - operand5; j++)
for (i = 0; i < c; i++)
if (operand2[j])
operand0[i][j] = operand1[j * c + i];
@@ -7393,7 +7393,7 @@ The operation is equivalent to:
@smallexample
int c = GET_MODE_SIZE (@var{m}) / GET_MODE_SIZE (@var{n});
-for (j = 0; j < operand3 + operand4; j++)
+for (j = 0; j < operand3 - operand4; j++)
if (operand2[j])
for (i = 0; i < c; i++)
operand0[j * c + i] = operand1[i][j];
@@ -7440,11 +7440,11 @@ Like @samp{gather_load@var{m}@var{n}}, but takes an extra mask operand
(operand 7) and a bias operand (operand 8).
Similar to mask_len_load the instruction loads at
-most (operand 7 + operand 8) elements from memory.
+most (operand 7 - operand 8) elements from memory.
Bit @var{i} of the mask is set if element @var{i} of the result should
be loaded from memory and clear if element @var{i} of the result should
be set to element @var{i} of operand 6.
-Mask elements @var{i} with @var{i} > (operand 7 + operand 8) are ignored.
+Mask elements @var{i} with @var{i} > (operand 7 - operand 8) are ignored.
@mdindex mask_len_strided_load@var{m}
@item @samp{mask_len_strided_load@var{m}}
@@ -7455,10 +7455,10 @@ operand 3 is mask operand, operand 4 is length operand and operand 5 is bias ope
The instruction can be seen as a special case of @code{mask_len_gather_load@var{m}@var{n}}
with an offset vector that is a @code{vec_series} with zero as base and operand 2 as step.
For each element the load address is operand 1 + @var{i} * operand 2.
-Similar to mask_len_load, the instruction loads at most (operand 4 + operand 5) elements from memory.
+Similar to mask_len_load, the instruction loads at most (operand 4 - operand 5) elements from memory.
Element @var{i} of the mask (operand 3) is set if element @var{i} of the result should
be loaded from memory and clear if element @var{i} of the result should be zero.
-Mask elements @var{i} with @var{i} > (operand 4 + operand 5) are ignored.
+Mask elements @var{i} with @var{i} > (operand 4 - operand 5) are ignored.
@mdindex scatter_store@var{m}@var{n}
@item @samp{scatter_store@var{m}@var{n}}
@@ -7497,9 +7497,9 @@ of the result should be stored to memory.
@item @samp{mask_len_scatter_store@var{m}@var{n}}
Like @samp{scatter_store@var{m}@var{n}}, but takes an extra mask operand (operand 5),
a len operand (operand 6) as well as a bias operand (operand 7). The instruction stores
-at most (operand 6 + operand 7) elements of (operand 4) to memory.
+at most (operand 6 - operand 7) elements of (operand 4) to memory.
Bit @var{i} of the mask is set if element @var{i} of (operand 4) should be stored.
-Mask elements @var{i} with @var{i} > (operand 6 + operand 7) are ignored.
+Mask elements @var{i} with @var{i} > (operand 6 - operand 7) are ignored.
@mdindex mask_len_strided_store@var{m}
@item @samp{mask_len_strided_store@var{m}}
@@ -7510,9 +7510,9 @@ operand 3 is mask operand, operand 4 is length operand and operand 5 is bias ope
The instruction can be seen as a special case of @code{mask_len_scatter_store@var{m}@var{n}}
with an offset vector that is a @code{vec_series} with zero as base and operand 1 as step.
For each element the store address is operand 0 + @var{i} * operand 1.
-Similar to mask_len_store, the instruction stores at most (operand 4 + operand 5) elements of
+Similar to mask_len_store, the instruction stores at most (operand 4 - operand 5) elements of
mask (operand 3) to memory. Element @var{i} of the mask is set if element @var{i} of (operand 3)
-should be stored. Mask elements @var{i} with @var{i} > (operand 4 + operand 5) are ignored.
+should be stored. Mask elements @var{i} with @var{i} > (operand 4 - operand 5) are ignored.
@mdindex while_ult@var{m}@var{n}
@item @code{while_ult@var{m}@var{n}}
@@ -7725,7 +7725,7 @@ This pattern is not allowed to @code{FAIL}.
@mdindex len_load_@var{m}
@item @samp{len_load_@var{m}}
-Load (operand 3 + operand 4) elements from memory operand 1
+Load (operand 3 - operand 4) elements from memory operand 1
into vector register operand 0. Operands 0 and 1 have mode @var{m},
which must be a vector mode. Operand 3 has whichever integer mode the
target prefers. Operand 2 (the ``else value'') is of mode @var{m} and
@@ -7738,7 +7738,7 @@ constant bias: it is either a constant 0 or a constant -1. The predicate on
operand 4 must only accept the bias values that the target actually supports.
GCC handles a bias of 0 more efficiently than a bias of -1.
-If (operand 3 + operand 4) exceeds the number of elements in mode
+If (operand 3 - operand 4) exceeds the number of elements in mode
@var{m}, the behavior is undefined.
If the target prefers the length to be measured in bytes rather than
@@ -7749,7 +7749,7 @@ This pattern is not allowed to @code{FAIL}.
@mdindex len_store_@var{m}
@item @samp{len_store_@var{m}}
-Store (operand 2 + operand 3) vector elements from vector register operand 1
+Store (operand 2 - operand 3) vector elements from vector register operand 1
into memory operand 0, leaving the other elements of
operand 0 unchanged. Operands 0 and 1 have mode @var{m}, which must be
a vector mode. Operand 2 has whichever integer mode the target prefers.
@@ -7760,7 +7760,7 @@ constant bias: it is either a constant 0 or a constant -1. The predicate on
operand 3 must only accept the bias values that the target actually supports.
GCC handles a bias of 0 more efficiently than a bias of -1.
-If (operand 2 + operand 3) exceeds the number of elements in mode
+If (operand 2 - operand 3) exceeds the number of elements in mode
@var{m}, the behavior is undefined.
If the target prefers the length to be measured in bytes
@@ -7772,23 +7772,23 @@ This pattern is not allowed to @code{FAIL}.
@mdindex mask_len_load@var{m}@var{n}
@item @samp{mask_len_load@var{m}@var{n}}
Perform a masked load from the memory location pointed to by operand 1
-into register operand 0. (operand 3 + operand 4) elements are loaded from
+into register operand 0. (operand 4 - operand 5) elements are loaded from
memory and other elements in operand 0 are set to undefined values.
This is a combination of len_load and maskload.
Operands 0 and 1 have mode @var{m}, which must be a vector mode. Operand 3
has whichever integer mode the target prefers. A mask is specified in
operand 2 which must be of type @var{n}. The mask has lower precedence than
the length and is itself subject to length masking,
-i.e. only mask indices < (operand 4 + operand 5) are used.
+i.e. only mask indices < (operand 4 - operand 5) are used.
Operand 3 is an else operand similar to the one in @code{maskload}.
-Operand 4 conceptually has mode @code{QI}.
+Operand 5 conceptually has mode @code{QI}.
Operand 4 can be a variable or a constant amount. Operand 5 specifies a
constant bias: it is either a constant 0 or a constant -1. The predicate on
operand 5 must only accept the bias values that the target actually supports.
GCC handles a bias of 0 more efficiently than a bias of -1.
-If (operand 4 + operand 5) exceeds the number of elements in mode
+If (operand 4 - operand 5) exceeds the number of elements in mode
@var{m}, the behavior is undefined.
If the target prefers the length to be measured in bytes
@@ -7800,13 +7800,13 @@ This pattern is not allowed to @code{FAIL}.
@mdindex mask_len_store@var{m}@var{n}
@item @samp{mask_len_store@var{m}@var{n}}
Perform a masked store from vector register operand 1 into memory operand 0.
-(operand 3 + operand 4) elements are stored to memory
+(operand 3 - operand 4) elements are stored to memory
and leave the other elements of operand 0 unchanged.
This is a combination of len_store and maskstore.
Operands 0 and 1 have mode @var{m}, which must be a vector mode. Operand 3 has whichever
integer mode the target prefers. A mask is specified in operand 2 which must be
of type @var{n}. The mask has lower precedence than the length and is itself subject to
-length masking, i.e. only mask indices < (operand 3 + operand 4) are used.
+length masking, i.e. only mask indices < (operand 3 - operand 4) are used.
Operand 4 conceptually has mode @code{QI}.
Operand 2 can be a variable or a constant amount. Operand 3 specifies a
@@ -7814,7 +7814,7 @@ constant bias: it is either a constant 0 or a constant -1. The predicate on
operand 4 must only accept the bias values that the target actually supports.
GCC handles a bias of 0 more efficiently than a bias of -1.
-If (operand 2 + operand 4) exceeds the number of elements in mode
+If (operand 3 - operand 4) exceeds the number of elements in mode
@var{m}, the behavior is undefined.
If the target prefers the length to be measured in bytes
@@ -8561,13 +8561,13 @@ for (i = 0; i < GET_MODE_NUNITS (@var{m}); i++)
@itemx @samp{cond_len_floor@var{mode}}
@itemx @samp{cond_len_round@var{mode}}
@itemx @samp{cond_len_rint@var{mode}}
-When operand 1 is true and element index < operand 4 + operand 5, perform an operation on operands 1 and
+When operand 1 is true and element index < operand 4 - operand 5, perform an operation on operands 1 and
store the result in operand 0, otherwise store operand 2 in operand 0.
The operation only works for the operands are vectors.
@smallexample
for (i = 0; i < GET_MODE_NUNITS (@var{m}); i++)
- op0[i] = (i < ops[4] + ops[5] && op1[i]
+ op0[i] = (i < ops[4] - ops[5] && op1[i]
? @var{op} op2[i]
: op3[i]);
@end smallexample
@@ -8627,13 +8627,13 @@ form of @samp{@var{op}@var{mode}2}.
@itemx @samp{cond_len_ashl@var{mode}}
@itemx @samp{cond_len_ashr@var{mode}}
@itemx @samp{cond_len_lshr@var{mode}}
-When operand 1 is true and element index < operand 5 + operand 6, perform an operation on operands 2 and 3 and
+When operand 1 is true and element index < operand 5 - operand 6, perform an operation on operands 2 and 3 and
store the result in operand 0, otherwise store operand 4 in operand 0.
The operation only works for the operands are vectors.
@smallexample
for (i = 0; i < GET_MODE_NUNITS (@var{m}); i++)
- op0[i] = (i < ops[5] + ops[6] && op1[i]
+ op0[i] = (i < ops[5] - ops[6] && op1[i]
? op2[i] @var{op} op3[i]
: op4[i]);
@end smallexample
@@ -8670,7 +8670,7 @@ takes 3 operands rather than two. For example, the vector form of
@smallexample
for (i = 0; i < GET_MODE_NUNITS (@var{m}); i++)
- op0[i] = (i < ops[6] + ops[7] && op1[i]
+ op0[i] = (i < ops[6] - ops[7] && op1[i]
? fma (op2[i], op3[i], op4[i])
: op5[i]);
@end smallexample
diff --git a/gcc/gimple-fold.cc b/gcc/gimple-fold.cc
index 17b65e0a9e92..e3df0d1163c7 100644
--- a/gcc/gimple-fold.cc
+++ b/gcc/gimple-fold.cc
@@ -5984,7 +5984,7 @@ partial_load_store_mask_state (gcall *call, tree vectype)
if (len && poly_int_tree_p (len))
{
gcc_assert (TREE_CODE (bias) == INTEGER_CST);
- wlen = wi::to_poly_widest (len) + wi::to_widest (bias);
+ wlen = wi::to_poly_widest (len) - wi::to_widest (bias);
if (known_eq (wlen, 0))
return MASK_ALL_INACTIVE;
diff --git a/gcc/testsuite/gfortran.dg/pr127100.f90 b/gcc/testsuite/gfortran.dg/pr127100.f90
new file mode 100644
index 000000000000..177017b07fbc
--- /dev/null
+++ b/gcc/testsuite/gfortran.dg/pr127100.f90
@@ -0,0 +1,11 @@
+! PR tree-optimization/127100
+! { dg-do compile { target s390*-*-* } }
+! { dg-options "-O2 -march=z13 -fdump-tree-optimized" }
+! Verify the test isn't miscompiled into unconditional STOP 1.
+! { dg-final { scan-tree-dump "return 0;" "optimized" } }
+
+program main
+ integer(kind=1), dimension(2,2) :: a
+ a = reshape((/ 1_1, 2_1, 3_1, 4_1/), shape(a))
+ if (any(matmul(a,a) /= reshape ( (/ 7, 10, 15, 22 /), shape(a)))) STOP 1
+end program main
More information about the Gcc-cvs
mailing list