[Bug target/89523] New: Incorrect AVX instructions with VSIB address

[hjl.tools at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89523

            Bug ID: 89523
           Summary: Incorrect AVX instructions with VSIB address
           Product: gcc
           Version: 7.4.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: target
          Assignee: unassigned at gcc dot gnu.org
          Reporter: hjl.tools at gmail dot com
                CC: ubizjak at gmail dot com
  Target Milestone: ---
            Target: x32

32-bit indices in VSIB address are sign-extended to 64 bits. In x32,
when 32-bit indices are used as addresses, like

%ymm7, 0(,%ymm6,1), %ymm0

0xf7fa3010 is sign-extended to 0xfffffffff7fa3010 which leads to
invalid address in x32.
Testcase:

[hjl@gnu-4 00000001]$ cat foo.i
void foo (void);

extern float *ncost;

float
bar (int type, int num)
{
  int i;
  float cost;

  cost = 0;
  for (i = 0; i < num; i++)
    if (type)
      cost += ncost[i];
    else
      foo ();
  return (cost);
}
[hjl@gnu-4 00000001]$ gcc -S -mx32 -Ofast -funroll-loops -march=haswell foo.i 
[hjl@gnu-4 00000001]$ grep gather foo.s
        vgatherdps      %ymm7, 0(,%ymm6,1), %ymm0
        vgatherdps      %ymm11, 0(,%ymm10,1), %ymm12
        vgatherdps      %ymm15, 0(,%ymm14,1), %ymm5
        vgatherdps      %ymm7, 0(,%ymm9,1), %ymm6
        vgatherdps      %ymm12, 0(,%ymm11,1), %ymm13
        vgatherdps      %ymm5, 0(,%ymm15,1), %ymm2
        vgatherdps      %ymm7, 0(,%ymm10,1), %ymm6
        vgatherdps      %ymm14, 0(,%ymm13,1), %ymm15
        vgatherdps      %ymm7, 0(,%ymm10,1), %ymm6
        vgatherdps      %ymm13, 0(,%ymm12,1), %ymm14
        vgatherdps      %ymm10, 0(,%ymm9,1), %ymm7
        vgatherdps      %ymm12, 0(,%ymm11,1), %ymm13
        vgatherdps      %ymm9, 0(,%ymm2,1), %ymm10
        vgatherdps      %ymm12, 0(,%ymm11,1), %ymm6
        vgatherdps      %ymm5, 0(,%ymm15,1), %ymm2
[hjl@gnu-4 00000001]$