[RFC] Pointer overflow produced by pointer_int_sum and PR26763

[Richard Guenther <rguenther at suse dot de>]

PR26763 is

int try (int *a)
{
  return a + -1 > a;
}

int main(void)
{
  int bla[100];
  if (try (bla + 50))
    abort ();
  return 0;
}

now, this looks like a perfectly valid testcase which pointer_int_sum
makes questionable, because it produces

  a + -4B > a

which has an overflowing pointer on the lhs.  I think we should avoid
this situation by simply producing a - 4B instead.  Does this sound
reasonable?  This is what the following (untested) patch does.

Opinions?

Thanks,
Richard.

Index: tree.h
===================================================================
*** tree.h	(revision 112577)
--- tree.h	(working copy)
*************** extern tree build_fold_indirect_ref (tre
*** 4196,4201 ****
--- 4196,4202 ----
  extern tree fold_indirect_ref (tree);
  extern tree constant_boolean_node (int, tree);
  extern tree build_low_bits_mask (tree, unsigned);
+ extern tree fold_negate_const (tree arg0, tree type);
  
  extern bool tree_swap_operands_p (tree, tree, bool);
  extern void swap_tree_operands (tree, tree *, tree *);
Index: fold-const.c
===================================================================
*** fold-const.c	(revision 112577)
--- fold-const.c	(working copy)
*************** static tree fold_mathfn_compare (enum bu
*** 131,137 ****
  static tree fold_inf_compare (enum tree_code, tree, tree, tree);
  static tree fold_div_compare (enum tree_code, tree, tree, tree);
  static bool reorder_operands_p (tree, tree);
! static tree fold_negate_const (tree, tree);
  static tree fold_not_const (tree, tree);
  static tree fold_relational_const (enum tree_code, tree, tree, tree);
  
--- 131,137 ----
  static tree fold_inf_compare (enum tree_code, tree, tree, tree);
  static tree fold_div_compare (enum tree_code, tree, tree, tree);
  static bool reorder_operands_p (tree, tree);
! tree fold_negate_const (tree, tree);
  static tree fold_not_const (tree, tree);
  static tree fold_relational_const (enum tree_code, tree, tree, tree);
  
*************** fold_read_from_constant_string (tree exp
*** 11822,11828 ****
  
     TYPE is the type of the result.  */
  
! static tree
  fold_negate_const (tree arg0, tree type)
  {
    tree t = NULL_TREE;
--- 11822,11828 ----
  
     TYPE is the type of the result.  */
  
! tree
  fold_negate_const (tree arg0, tree type)
  {
    tree t = NULL_TREE;
Index: c-common.c
===================================================================
*** c-common.c	(revision 112577)
--- c-common.c	(working copy)
*************** pointer_int_sum (enum tree_code resultco
*** 2354,2359 ****
--- 2354,2368 ----
    else
      size_exp = size_in_bytes (TREE_TYPE (result_type));
  
+   /* If intop is negative, invert the resultcode to avoid relying
+      on pointer overflow being defined.  */
+   if (TREE_CODE (intop) == INTEGER_CST
+       && tree_int_cst_sgn (intop) < 0)
+     {
+       resultcode = resultcode == PLUS_EXPR ? MINUS_EXPR : PLUS_EXPR;
+       intop = fold_negate_const (intop, TREE_TYPE (intop));
+     }
+ 
    /* If what we are about to multiply by the size of the elements
       contains a constant term, apply distributive law
       and multiply that constant term separately.