fix allocno comparison function misbehavior on overflow

[Olivier Hainque <hainque at adacore dot com>]

Hello,

Compiling a big proprietary Ada source base with gcc 4.5 on x86-solaris
exposed misbehavior like

    received signal SIGSEGV, Segmentation fault.
    in allocno_priority_compare_func (v1p=0x21b0ce4, v2p=0x21b0cec)
    at ../../src/gcc/ira-color.c:1745
    1745      pri1 = allocno_priorities[ALLOCNO_NUM (a1)];

Very similar to what was reported as PR/44763 on existing testcases.

This turned out to be caused by a glitch with respect to possible overflows in
allocno_priority_compare_func:

  <<  pri1 = allocno_priorities[ALLOCNO_NUM (a1)];
      pri2 = allocno_priorities[ALLOCNO_NUM (a2)];
      if (pri2 - pri1)
        return pri2 - pri1;
  >>

The case we had ended up here with

  p2 = 247786560, p1 = -1982292480

so p2 > p1, but the function computed otherwise (p2-p1 = -2064888256).

This, in turn, lead to inconsistencies in the ordering relationship,
which the x86-solaris qsort implementation dislikes.

The attached patch is a proposal to address this by replacing the difference
computation by a comparison, factored out in a macro to allow reuse in other
places (not in this patch) and provide a clean spot for a general comment.

This fixed the observed SEGV with 4.5 and bootstrapped on i386-solaris. Was
also bootstrapped and regression tested for mainline on both sparc-solaris
and x86_64-linux.

OK ?

Thanks in advance,

With Kind Regards,

Olivier

2010-09-01  Olivier Hainque  <hainque@adacore.com>

        * ira-color.c (SORTGT): New macro, helper for qsort callbacks.
        (allocno_priority_compare_func): Use it instead of a straight
        difference computation over priorities.

Attachment: regsort.dif
Description: Text document