x86-64 abi, regarding vector types

[Richard Henderson <rth at redhat dot com>]

Since gcc 3.3 or so, gcc has attempted to support vector types
on all targets, with varying degrees of success.  For gcc 4.0,
this support was rewritten, and in the process is exposing ABI
incompatibility problems between gcc versions.

In the case of x86-64, the published abi is under-specified,
only mentioning the __m64 and __m128 types defined in the Intel
mmintrin.h and xmmintrin.h headers.

What isn't specified is what happens with other vector types.

By way of explanation, the syntax since gcc 3.4 for defining
a vector type is

	typedef TYPE name __attribute__((vector_size (SIZE)));

You can apply this to a data declaration as well instead of
indirectly by way of a typedef.  The semantics of this is that
TYPE must be a scalar, and SIZE must a power-of-2 multiple of
sizeof(TYPE).  For instance,

	typedef int signed_v4si __attribute__((vector_size (16)));

creates a 4 wide vector of signed integers.

The problem is that it's not *clear* that signed_v4si should be
treated like __m128 for the purposes of calling conventions.

It so happens that gcc 3.4 treats all 16 byte vectors as class SSE,
and passes all 8 byte in memory, and crashes when asked to pass
anything else.  Which does mean that gcc 3.4 is *incorrect* in its
handling of __m64, and should be fixed.  Since __m64 must be treated
as class SSE, I intend to treat all other 8 byte vector types the
same way.  Which includes the 3DNOW 2-wide float type.

For any vector type the user creates that's larger than 16 bytes, 
it's clear that should be passed in memory because of the size
limit in clause 1 of aggregate classification.

However, that leaves, exhaustively, the 2 and 4 byte vector types

	typedef char v2qi __attribute__((vector_size (2)));
	typedef char v4hi __attribute__((vector_size (4)));
	typedef short v2hi __attribute__((vector_size (4)));

My thought is that these should be treated the same as if they
were an array in the classification phase, which would put them
squarely in class INTEGER.

Comments?  Suggestions for the exact wording to add to the ABI?


r~