[PATCH] Fix qsort ordering violation in tree-vrp.c

[Richard Biener]

On Sat, 15 Jul 2017, Yuri Gribov wrote:

> Hi,
> 
> This is a follow-up on https://gcc.gnu.org/ml/gcc/2017-07/msg00078.html
> 
> compare_assert_loc in tree-vrp.c could return unpredictable results
> due to implicit conversion of unsigned subtraction to int here:
>   return ha - hb;
> 
> This could return inconsistent results for objects with the following hashes:
>   a: 0xdc8e4f72U
>   b: 0x912ab538U
>   c: 0x5ae66e3bU
> Then
>   a > b because (int)(0xdc8e4f72U - 0x912ab538U) > 0
>   b > c because (int)(0x912ab538U - 0x5ae66e3bU) > 0
> but
>   a < c because (int)(0xdc8e4f72U - 0x5ae66e3bU) == (int)0x81a7e137U < 0
> 
> Bug was found with https://github.com/yugr/sortcheck
> 
> Bootstrapped and regtested in x64, ok for trunk?

Ok.

Thanks,
Richard.