[PATCH, rtl-optimization]: Fix PR37997: ICE shifting byte to the right with constant > 7FFFFFFF

[Joseph S. Myers]

On Wed, 9 Sep 2009, Eric Botcazou wrote:

> > 0x7fffffff is signed, 0x80000000 unsigned; other than that, no.
> 
> OK, the (int) v1 >> 0x80000000 is shortened to "char" in build_binary_op:
> 
> 	  if (TYPE_PRECISION (TREE_TYPE (arg0)) < TYPE_PRECISION (result_type)
> 	      /* We can shorten only if the shift count is less than the
> 		 number of bits in the smaller type size.  */
> 	      && compare_tree_int (op1, TYPE_PRECISION (TREE_TYPE (arg0))) < 0
> 	      /* We cannot drop an unsigned shift after sign-extension.  */
> 	      && (!TYPE_UNSIGNED (final_type) || unsigned_arg))
> 	    {
> 	      /* Do an unsigned shift if the operand was zero-extended.  */
> 	      result_type
> 		= c_common_signed_or_unsigned_type (unsigned_arg,
> 						    TREE_TYPE (arg0));
> 	      /* Convert value-to-be-shifted to that type.  */
> 	      if (TREE_TYPE (op0) != result_type)
> 		op0 = convert (result_type, op0);
> 	      converted = 1;
> 	    }
> 
> because
> 
>   compare_tree_int (op1, TYPE_PRECISION (TREE_TYPE (arg0))) < 0
> 
> return true since op1 is -2147483648.
> 
> It seems to me that the problem should be fixed there.

I fail to see any problem.  build_binary_op earlier converted all shift 
counts to type int, but there is no problem with that since it will only 
change the value in cases of undefined behavior and you need to handle 
shifts by INT_MIN anyway.  There is no indication I can see in anything 
you have posted in this dicussion that the front end is generating any 
invalid trees or GIMPLE; it's for the expanders to handle arbitrary 
constant or nonconstant shift counts whose type may be unrelated to the 
type of the expression being shifted (where if it is known that the count 
is negative or >= width of type, the result is arbitrary).

-- 
Joseph S. Myers
joseph@codesourcery.com