[Bug tree-optimization/80072] [7 Regression] ICE in gimple_build_assign_1 with -O3 -march=broadwell/skylake-avx512

[rguenther at suse dot de]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80072

--- Comment #8 from rguenther at suse dot de <rguenther at suse dot de> ---
On Wed, 22 Mar 2017, jakub at gcc dot gnu.org wrote:

> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80072
> 
> --- Comment #7 from Jakub Jelinek <jakub at gcc dot gnu.org> ---
> So, found a bug and its bugfix fixes the testcase, but still need to figure out
> what actually was going on, if it was just that we made (correct) assumptions
> that constants won't be weirdly sorted, or if there is some other (with this
> patch latent) bug.
> 
> --- gcc/tree-ssa-reassoc.c.jj   2017-02-10 09:46:50.000000000 +0100
> +++ gcc/tree-ssa-reassoc.c      2017-03-22 13:13:33.608745625 +0100
> @@ -510,7 +510,7 @@ sort_by_operand_rank (const void *pa, co
> 
>    /* Lastly, make sure the versions that are the same go next to each
>       other.  */
> -  if ((oeb->rank - oea->rank == 0)
> +  if (oeb->rank == oea->rank
>        && TREE_CODE (oea->op) == SSA_NAME
>        && TREE_CODE (oeb->op) == SSA_NAME)
>      {
> @@ -549,7 +549,7 @@ sort_by_operand_rank (const void *pa, co
>      }
> 
>    if (oeb->rank != oea->rank)
> -    return oeb->rank - oea->rank;
> +    return oeb->rank > oea->rank ? 1 : -1;
>    else
>      return oeb->id - oea->id;
>  }
> 
> ->rank is unsigned int, and on the huge testcase it is over 0x7fffffff, so
> (int) (oeb->rank > oea->rank) doesn't do what it expects and we happily sort
> a SSA_NAME with extra high rank after constants, which e.g. means we can't
> merge them together.

Ick ;)

Patch looks obvious.  For clarity the else return oeb->id - oea->id
should probably be adjusted similarly (and id made unsigned...).