[Patch, stage-1, RFC]: i386: attribute regparm/stdcall and vaargs

[Bernhard Reutner-Fischer]

[I was torn towards asking gcc@ only, individual i386 maintainers in
private or bluntly asking for help on gcc-patches or re-iterate through
ABI, so in an attempt to cut off years of latency i hereby ask all and
everybody for assistance. Stage4 means any chances are low, i know..
hence stage 1 material since it's not pressing in any foreseeable way]
Hello i386 maintainers

Recently, elsewhere, there was discussion about attribute regparm (and
stdcall) on functions with variable number of arguments in C.
Allegedly clang warns about these but GCC does not. I did not look at
clang.

gcc/doc/extend.texi currently states that:

---8<---
@cindex @code{regparm} function attribute, x86
[]
Functions that
take a variable number of arguments continue to be passed all of their
arguments on the stack.
[]
@cindex @code{sseregparm} function attribute, x86
[]
Functions that take a
variable number of arguments continue to pass all of their
floating-point arguments on the stack.
[]
@cindex @code{stdcall} function attribute, x86-32
[]
On x86-32 targets, the @code{stdcall} attribute causes the compiler to
assume that the called function pops off the stack space used to
pass arguments, unless it takes a variable number of arguments.
---8<---

which seems to suggest that all of attribute regparm/sseregparm/stdcall
are ignored on functions with variable number of arguments. I.e. the
ABI mandates that everything is passed on the stack.
[Right? ISTM that this is correct; Didn't follow ABI (tweaks) too
closely in the last decade, admittedly.. But i think this still holds.
Please correct me if i missed something?]

If this is correct, then ISTM that attributes
regparm/sseregparm/stdcall should be rejected on functions with
variable number of arguments also in GCC.

There seems to be (exact) struct function cfun->va_list_[fg]pr_size
for the real fpr and gpr save area sizes. But (unfortunately / of
course) they are layed out way later than parsing the attributes in
both the C++ and C FEs, so using those in ix86_handle_cconv_attribute
is not possible as there is no cfun readily available there yet. ²).

Hence i would propose to ¹):

gcc/ChangeLog:

	* builtin-attrs.def (ATTR_TM_NOTHROW_RT_LIST): Use ATTR_NOTHROW_LIST
	instead of ATTR_TM_NOTHROW_LIST, thus removing ATTR_TM_REGPARM.
	* config/i386/i386-options.cc (ix86_handle_cconv_attribute): Decline
	regparm, stdcall and regparm attribute on functions with variable number
	of arguments.

libitm/ChangeLog:

	* libitm.h (_ITM_beginTransaction): Remove ITM_REGPARM.

gcc/testsuite/ChangeLog:

	* gcc.dg/lto/trans-mem.h: Remove ITM_REGPARM.
	* gcc.target/i386/attributes-error.c: Extend to cover
	(regparm(3),stdcall) on vaargs functions.
	* gcc.target/i386/attributes-error-sse.c: New test.

¹) as per attached
²) Unfortunately, the C FE does not readily provide a sensible locus
for the attributes in question but points input_location at that spot
of the beginning of the declaration of such a function. The C++ FE is
a little bit better in this regard:
[here i meant to verbatim emphasis discrepancy of the C++ and C FE
diagnostics for the abovementioned target tests, striking, isn't it, But
see yourselves.]
³) unreferenced, hence implied, where would on do this instead, more
helpful?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fwd-i386-attrib-vs-vaargs.00.patch
Type: text/x-patch
Size: 5034 bytes
Desc: not available
URL: <https://gcc.gnu.org/pipermail/gcc/attachments/20240130/4651b398/attachment-0001.bin>