rs6000 PPC32 SVR4 ABI long double function return bug

[Jim Wilson]

The Motorola SPE testsuite found another SVR4 PPC32 ABI problem.

long doubles are supposed to be returned via memory.  Gcc is returning them
in registers.

I am assuming that the current behaviour is correct for AIX.  I tried to check,
but only managed to find one AIX machine, and it wouldn't let me log in.

I am assuming that ports that are currently incompatible with the SVR4
structure return conventions want to continue using the current conventions
to avoid ABI incompatibilities.  This includes linux.

I am assuming that ports that are currently compatible with the SVR4
stucture return conventions want the incompatible ABI change to remain
compatible with the standard.  This includes freebsd and netbsd.

I am assuming that no user would want to change structure passing conventions
via command line options.

I don't have copies of any PPC psABI draft, so I can't tell if this changed
between the draft and the final edition.  I am assuming that we just forgot
to implement this part of the long double support when we added 128-bit long
double support.

This gives me the following patch.  I'd like comment from some of the rs6000
port maintainers, since I am not familiar with all of the PPC ABIs, and I'm
not sure if all of my assumptions are correct.  I'd be happy to change any part
of this patch if you want this implemented differently.

With this patch, the ABI tests in the Motorola SPE testsuite all seem to pass
compiled for big-endian.  My full testsuite run is still going, but it is
looking good so far.
	
I noticed the PPC LSB states that function return values follow SVR4
conventions except for structs and unions.  If we choose not to fix the
linux port, then the PPC LSB will have to be modified to note that long
double return values are also handled differently.

2002-12-05  Jim Wilson  <wilson@redhat.com>

	* linux.h (V4_LONG_DOUBLE_RET): Define.
	* rs6000.h (RETURN_IN_MEMORY): If ABI_V4 and V4_LONG_DOUBLE_RET then
	TFmode is returned in memory.
	(V4_LONG_DOUBLE_RET): Define.

Index: linux.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/config/rs6000/linux.h,v
retrieving revision 1.34
diff -p -r1.34 linux.h
*** linux.h	10 Nov 2002 20:47:44 -0000	1.34
--- linux.h	6 Dec 2002 03:07:58 -0000
*************** Boston, MA 02111-1307, USA.  */
*** 79,84 ****
--- 79,89 ----
  #undef DRAFT_V4_STRUCT_RET
  #define DRAFT_V4_STRUCT_RET 1
  
+ /* For backward compatibility, we must continue to use the AIX
+    long double return convention.  */
+ #undef V4_LONG_DOUBLE_RET
+ #define V4_LONG_DOUBLE_RET 0
+ 
  /* Do code reading to identify a signal frame, and set the frame
     state data appropriately.  See unwind-dw2.c for the structs.  */
  
Index: rs6000.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/config/rs6000/rs6000.h,v
retrieving revision 1.235
diff -p -r1.235 rs6000.h
*** rs6000.h	25 Nov 2002 19:40:01 -0000	1.235
--- rs6000.h	6 Dec 2002 03:07:59 -0000
*************** typedef struct rs6000_stack {
*** 1530,1545 ****
     default, and -m switches get the final word.  See
     rs6000_override_options for more details.
  
     int_size_in_bytes returns -1 for variable size objects, which go in
     memory always.  The cast to unsigned makes -1 > 8.  */
  
  #define RETURN_IN_MEMORY(TYPE) \
!   (AGGREGATE_TYPE_P (TYPE) && \
!    (TARGET_AIX_STRUCT_RET || \
!     (unsigned HOST_WIDE_INT) int_size_in_bytes (TYPE) > 8))
  
  /* DRAFT_V4_STRUCT_RET defaults off.  */
  #define DRAFT_V4_STRUCT_RET 0
  
  /* Let RETURN_IN_MEMORY control what happens.  */
  #define DEFAULT_PCC_STRUCT_RETURN 0
--- 1530,1554 ----
     default, and -m switches get the final word.  See
     rs6000_override_options for more details.
  
+    The SYSV.4 ABI says long double is returned in memory.  This was wrong
+    in the initial implementation, so we allow a target to get the old
+    behaviour by changing V4_LONG_DOUBLE_RET.
+ 
     int_size_in_bytes returns -1 for variable size objects, which go in
     memory always.  The cast to unsigned makes -1 > 8.  */
  
  #define RETURN_IN_MEMORY(TYPE) \
!   ((AGGREGATE_TYPE_P (TYPE)					\
!     && (TARGET_AIX_STRUCT_RET ||				\
! 	(unsigned HOST_WIDE_INT) int_size_in_bytes (TYPE) > 8))	\
!    || (DEFAULT_ABI == ABI_V4 && V4_LONG_DOUBLE_RET		\
!        && TYPE_MODE (TYPE) == TFmode))
  
  /* DRAFT_V4_STRUCT_RET defaults off.  */
  #define DRAFT_V4_STRUCT_RET 0
+ 
+ /* V4_LONG_DOUBLE_RET defaults on.  */
+ #define V4_LONG_DOUBLE_RET 1
  
  /* Let RETURN_IN_MEMORY control what happens.  */
  #define DEFAULT_PCC_STRUCT_RETURN 0