[x86 PATCH]: RETURN_IN_MEMORY vs. i386's type_natural_mode

[Roger Sayle <roger at eyesopen dot com>]

Either (or both) of the following patches resolve 20 unexpected failures
on i386-pc-solaris2.10, including gcc.c-torture/execute/simd-6.c.
As simd-6.c compiles without problems using gcc 3.4.3, this is an
ICE on valid regression from a previous release.

The issue concerns the x86 backend's handling of ABIs that return vector
modes such as V8QI or V2SI on CPU targets without MMX (likewise SSE).
Conceptually, a subtarget has one of two choices, it can either force
the result to be returned in memory, or it can treat an 8 byte result
as it would a DImode result, and return it in eax:edx.


Unfortunately, the core x86 backend doesn't currently support this
latter strategy to efficiently return results in registers when MMX/SSE
isn't available.  The code in ix86_value_regno contains the lines:

  /* 8-byte vector modes in %mm0. See ix86_return_in_memory for where
     we prevent this case when mmx is not available.  */
  if ((VECTOR_MODE_P (mode) && GET_MODE_SIZE (mode) == 8))
    return FIRST_MMX_REG;

Unfortunately, several x86 subtargets, including svr4 and solaris 2.10,
provide their own definitions of RETURN_IN_MEMORY which doesn't always
match the logic in ix86_return_in_memory.  This results in a ICE from
an unrecognizable insn as we attempt to return a DImode result in
FIRST_MMX_REG, which isn't a valid hard register.

There are two sides to every argument, so naturally two fixes.

The first patch below tweaks i386.c's ix86_value_regno so it can
support returning vectors without hadware support in registers in
addition to in memory.  Much like a middle-end change, this allows
x86 subtargets to decide for themselves what they want to do.  This
is also an "obvious" sanity fix, as it's never correct to refer
to FIRST_MMX_REG when !TARGET_MMX.  If we were being difficult, we
could just add an assert to fail earlier, but for little extra
effort seems no harm in allowing ABI designers some flexibility.


The second patch below addresses the potential issue that asking the
subtarget-independent x86 backend to return these vectors in registers
may be an unintentional mistake in i386/sol2-10.h, which is based upon
svr4's ABI, which does the same thing (i386/sysv4.h contains the same
logic error).

The problem here is that the type that gets passed to RETURN_IN_MEMORY
has a TYPE_MODE of DImode when MMX isn't available.  This clearly fails
the VECTOR_MODE_P check, and a DImode doesn't normally get returned in
memory.  Indeed, the mismatch is that underneath the surface the x86
backend internally uses "natural_type_mode" instead of TYPE_MODE, to
identify  exactly these corner cases.

The simple fix is to replace VECTOR_MODE_P (TYPE_MODE (t)) which
varies with the availability of MMX, with TREE_CODE (t) == VECTOR_TYPE.
This seems a simpler fix than exporting i386.c's natural_type_mode,
which is currently a local static function, and not currently
visible everywhere the RETURN_IN_MEMORY macro is used.

This, of course, is a potential ABI change.   And changing just
sol2-10.h means that it no longer follows the SVR4 implementation
in i386/sysv4.h.  Hence unlike the first patch to improve GCC's
functionality, this second fix is at the discretion of a backend
maintainer who understands what the intended/correct behaviour
is for SVR4 and/or Solaris/x86 ABIs.


Both of the following patches have been tested on i386-pc-solaris2.10,
with a full "make bootstrap", including C, C++, ObjC and gfortran,
and regression tested with a top-level "make -k check" with no new
failures, but with 20 less unexpected failures in the gcc testsuite
as mentioned above.  Although both cure the same ICE, individually
each patch implements a different ABI.

Thoughts and comments?



2006-07-25  Roger Sayle  <roger@eyesopen.com>

	* config/i386/i386.c (ix86_value_regno): Don't return FIRST_MMX_REG
	if !TARGET_MMX, and don't return FIRST_SSE_REG if !TARGET_SSE.


Index: config/i386/i386.c
===================================================================
*** config/i386/i386.c	(revision 115686)
--- config/i386/i386.c	(working copy)
*************** ix86_value_regno (enum machine_mode mode
*** 3980,3993 ****
    gcc_assert (!TARGET_64BIT);

    /* 8-byte vector modes in %mm0. See ix86_return_in_memory for where
!      we prevent this case when mmx is not available.  */
!   if ((VECTOR_MODE_P (mode) && GET_MODE_SIZE (mode) == 8))
!     return FIRST_MMX_REG;

    /* 16-byte vector modes in %xmm0.  See ix86_return_in_memory for where
       we prevent this case when sse is not available.  */
    if (mode == TImode || (VECTOR_MODE_P (mode) && GET_MODE_SIZE (mode) == 16))
!     return FIRST_SSE_REG;

    /* Decimal floating point values can go in %eax, unlike other float modes.  */
    if (DECIMAL_FLOAT_MODE_P (mode))
--- 3980,3994 ----
    gcc_assert (!TARGET_64BIT);

    /* 8-byte vector modes in %mm0. See ix86_return_in_memory for where
!      we normally prevent this case when mmx is not available.  However
!      some ABIs may require the result to returned like DImode.  */
!   if (VECTOR_MODE_P (mode) && GET_MODE_SIZE (mode) == 8)
!     return TARGET_MMX ? FIRST_MMX_REG : 0;

    /* 16-byte vector modes in %xmm0.  See ix86_return_in_memory for where
       we prevent this case when sse is not available.  */
    if (mode == TImode || (VECTOR_MODE_P (mode) && GET_MODE_SIZE (mode) == 16))
!     return TARGET_SSE ? FIRST_SSE_REG : 0;

    /* Decimal floating point values can go in %eax, unlike other float modes.  */
    if (DECIMAL_FLOAT_MODE_P (mode))



2006-07-25  Roger Sayle  <roger@eyesopen.com>

	* config/i386/sol2-10.h (RETURN_IN_MEMORY): Check whether the type
	is a vector type instead of whether the type's mode is a vector mode.


Index: config/i386/sol2-10.h
===================================================================
*** config/i386/sol2-10.h	(revision 115686)
--- config/i386/sol2-10.h	(working copy)
*************** Boston, MA 02110-1301, USA.  */
*** 110,114 ****
    (TARGET_64BIT 				\
     ? ix86_return_in_memory (TYPE)		\
     : (TYPE_MODE (TYPE) == BLKmode		\
!       || (VECTOR_MODE_P (TYPE_MODE (TYPE)) 	\
  	  && int_size_in_bytes (TYPE) == 8)))
--- 110,114 ----
    (TARGET_64BIT 				\
     ? ix86_return_in_memory (TYPE)		\
     : (TYPE_MODE (TYPE) == BLKmode		\
!       || (TREE_CODE (TYPE) == VECTOR_TYPE 	\
  	  && int_size_in_bytes (TYPE) == 8)))


Roger
--