[Bug c/87379] New: Warn about function pointer casts which differ in variadic-ness

[fw at gcc dot gnu.org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=87379

            Bug ID: 87379
           Summary: Warn about function pointer casts which differ in
                    variadic-ness
           Product: gcc
           Version: 9.0
            Status: UNCONFIRMED
          Keywords: diagnostic
          Severity: normal
          Priority: P3
         Component: c
          Assignee: unassigned at gcc dot gnu.org
          Reporter: fw at gcc dot gnu.org
                CC: msebor at gcc dot gnu.org
  Target Milestone: ---

It is fairly common to assume that that the ... part of a variadic argument
list is a wildcard that that functions and function pointers which agree on the
non-... parts are ABI-compatible.  However, this is wrong for x86-64 (where
%rax needs to be set up for variadic parts containing floating point arguments)
and the ELFv2 ABI for POWER (where a parameter save area must be created in the
caller for variadic calls).

So I would like to see a warning for this:

int open (const char *, int, ...);
int (*ptr1) (const char *, int) = (int (*) (const char *, int)) open;
int (*ptr2) (const char *, int, short)
  = (int (*) (const char *, int, short)) open;

But perhaps we should not warn for this because the variadic nature is
preserved in the non-prototyped function pointer:

void (*ptr3) () = (void (*) ()) open;

Whether we want to warn for this up to debate because these casts are clearly
different, so it is more likely that the programmer knows what they are doing:

void (*ptr4) (double) = (void (*) (double) open;

Since multiple targets are affected, I think it makes sense to warn for this
issue independently of the current target.

Calls with variadic-ness mismatches cause bugs which can be very difficult to
track down.  The bug I particularly remember is this one:

commit c7774174beffe9a8d29dd4fb38bbed43ece1cecd
Author: Andreas Schneider <asn@samba.org>
Date:   Wed Aug 2 13:21:59 2017 +0200

    swrap: Fix prototype of open[64] to prevent segfault on ppc64le

    The calling conventions for vaarg are different on ppc64le. The patch
    fixes segfaults on that platform.

<https://git.samba.org/?p=socket_wrapper.git;a=commitdiff;h=c7774174beffe9a8d29dd4fb38bbed43ece1cecd>

Here the problematic pointer came through dlopen, so a warning for casts would
not have helped.  It was difficult to diagnose because the lack of the
parameter save area (which was supposed by the called function to be allocated
by the caller) caused stack corruption a couple of frames up the stack, in a
fairly different area of the code.