[Bug other/18877] New: decompose (reload.c) can be forced to access hard_regno_nregs over its array bounds

["e9925248 at stud4 dot tuwien dot ac dot at" <gcc-bugzilla at gcc dot gnu dot org>]

true_regnum(x) returns only positive numbers for a pseudo register, if REG_P(x)  
is true.  
  
reload.c, function decompose, line 2363 (CVS HEAD version):  
  
   case REG:  
     val.reg_flag = 1;  
     val.start = true_regnum (x);  
      if (val.start < 0)  
        {  
          /* A pseudo with no hard reg.  */  
          val.start = REGNO (x);  
          val.end = val.start + 1;  
        }  
      else  
        /* A hard reg.  */  
        val.end = val.start + hard_regno_nregs[val.start][GET_MODE (x)];  
  
  
So if decompose is called with an pseudo register, which is not renumbered,  
val.end = val.start + hard_regno_nregs[val.start][GET_MODE (x)];  
is executed. val.start contains in that case a number >= FIRST_PSEUDO_REGISTER,  
which is so an access over the array-bounds of hard_regno_nregs.  
  
I suggest to put a   
gcc_assert (val.start <FIRST_PSEUDO_REGISTER);  
 before the instruction, so that such an error will be found.  
  
GCC calls decompose with a not renumbered register, if the following conditions  
are fullfiled:  
*All for allocation available registeres are in use  
*An operation has for an operand an constraint like "=&rm" and the value is an  
pseudo register.  
  
For i686 and gcc version 4.0.0 20041207 (experimental), such an assert fails   
for the following example (compiled with -O1,-O2,-O3 or -Os):  
long a,b,c,d,e,f,g,h,i;  
int test()  
{  
  long x1,x2,x3,l,l1,l2;  
  t1(&x1,&x2,&x3);  
  x1+=2;  
  l=c+2;  
  l1=d+2;  
  l2=e+2;  
  asm("nop # %0 %1 %2 %3 %4 %5 %6 %7 %8 %9": "=a"(a), "=b"(b), "=c"(c),  
 "=d"(d), "=&mr"(e), "=&mr"(f),"=&rm"(x2),"=&rm"(l2): "2"(x1),"0"(x2),  
"1"(e),"3"(f),"rm"(l),"rm"(l1),"rm"(l2),"m"(x3),"g"(x2));  
  t1(x1,x2,x3+l2);  
}  
  
In that case, hard_regno_nregs[58][SImode] will be accessed. If some code is  
inserted before the assignement of l2, l2 will get an higher number, which will  
cause an segmentation fault.  
  
As an fix, maybe changing the if-statement to  
if (val.start < 0||val.start>=FIRST_PSEUDO_REGISTER)  
can help, as the comment says, that the then-part of the if statement should be  
used for pseudo registers. If that is correct, I can't tell.  
  
For the SUBREG-case in decompose, there is a similar issue.  
  
mfg Martin Kögler  
e9925248@stud4.tuwien.ac.at

-- 
           Summary: decompose (reload.c) can be forced to access
                    hard_regno_nregs over its array bounds
           Product: gcc
           Version: 4.0.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: other
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: e9925248 at stud4 dot tuwien dot ac dot at
                CC: gcc-bugs at gcc dot gnu dot org
 GCC build triplet: i686-pc-linux-gnu (not i686 spezific)
  GCC host triplet: i686-pc-linux-gnu
GCC target triplet: i686-pc-linux-gnu


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=18877