[Bug tree-optimization/71289] Fails to optimize unsigned mul overflow check 'A > -1 / B'

[amonakov at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=71289

--- Comment #2 from Alexander Monakov <amonakov at gcc dot gnu.org> ---
> What do the dumps look like? Gcc is likely to change things to -1 / B < A, which you don't handle...

The dumps didn't help much, but you're right that normally the order is
opposite, thanks (I didn't realize that when looking in the debugger the first
time).

I tried to take into account your comments. I also had to make the output more
complex because emitting bare internal functions doesn't work. It still ICEs
for

typedef unsigned v2si __attribute__((vector_size(8)));
v2si h(v2si a, v2si b)
{
  v2si m = {~0, ~0};
  return (a > m / b);
}

but before debugging that I'd like to know if this is the right approach in the
first place.

diff --git a/gcc/match.pd b/gcc/match.pd
index e511e9a..a7fbcf1 100644
--- a/gcc/match.pd
+++ b/gcc/match.pd
@@ -2582,6 +2582,32 @@ DEFINE_INT_AND_FLOAT_ROUND_FN (RINT)
        && types_match (TREE_TYPE (@0), TREE_TYPE (@1)))
    (out (imagpart @2) { build_zero_cst (TREE_TYPE (@0)); }))))

+/* Simplify unsigned multiplication overflow check -1 / A < B to a builtin. 
*/
+(for cmp (lt ge)
+     out (ne eq)
+ (simplify
+  (cmp (trunc_div:s integer_all_onesp @0) @1)
+  (if (TYPE_UNSIGNED (TREE_TYPE (@0))
+       && !VECTOR_INTEGER_TYPE_P (TREE_TYPE (@0)))
+   (out
+    (imagpart { build_call_expr_internal_loc (UNKNOWN_LOCATION,
+                                             IFN_MUL_OVERFLOW,
+                                             build_complex_type (TREE_TYPE
(@0)),
+                                             2, @0, @1); } )
+    { build_zero_cst (TREE_TYPE (@0)); }))))
+
+(for cmp (gt le)
+     out (ne eq)
+ (simplify
+  (cmp @0 (trunc_div:s integer_all_onesp @1))
+  (if (TYPE_UNSIGNED (TREE_TYPE (@0))
+       && !VECTOR_INTEGER_TYPE_P (TREE_TYPE (@0)))
+   (out
+    (imagpart { build_call_expr_internal_loc (UNKNOWN_LOCATION,
+                                             IFN_MUL_OVERFLOW,
+                                             build_complex_type (TREE_TYPE
(@0)),
+                                             2, @0, @1); } )
+    { build_zero_cst (TREE_TYPE (@0)); }))))

 /* Simplification of math builtins.  These rules must all be optimizations
    as well as IL simplifications.  If there is a possibility that the new