combine: Fix up expand_compound_operation [PR99905]

combine: Fix up expand_compound_operation [PR99905]

The following testcase is miscompiled on x86_64-linux.
expand_compound_operation is called on
(zero_extract:DI (mem/c:TI (reg/f:DI 16 argp) [3 i+0 S16 A128])
    (const_int 16 [0x10])
    (const_int 63 [0x3f]))
so mode is DImode, inner_mode is TImode, pos 63, len 16 and modewidth 64.

A couple of lines above the problematic spot we have:
  if (modewidth >= pos + len)
    {
      tem = gen_lowpart (mode, XEXP (x, 0));
where the code uses gen_lowpart and then shift left/right to extract it
in mode.  But the guarding condition is false - 64 >= 63 + 16
and so we enter the next condition, where the code shifts XEXP (x, 0)
right by pos and then adds AND.  It does so incorrectly though.
Given the modewidth < pos + len, inner_mode must be necessarily larger
than mode and XEXP (x, 0) has the innermode, but it was calling
simplify_shift_const with mode rather than inner_mode, which meant
inconsistent arguments to simplify_shift_const and in this case made
a DImode MEM shift out of it.

The following patch fixes it, by doing the shift in inner_mode properly
and then after the shift doing the lowpart subreg and masking already
in mode.

2021-04-13  Jakub Jelinek  <jakub@redhat.com>

	PR rtl-optimization/99905
	* combine.c (expand_compound_operation): If pos + len > modewidth,
	perform the right shift by pos in inner_mode and then convert to mode,
	instead of trying to simplify a shift of rtx with inner_mode by pos
	as if it was a shift in mode.

	* gcc.target/i386/pr99905.c: New test.

(cherry picked from commit c965254e5af9dc68444e0289250c393ae0cd6131)