gcc 4.2 more strict check for "function called through a non-compatible type"

[Yuri Pudgorodsky <yur at ptci dot ru>]

Compiling openssl-0.9.8b with gcc-4.2 snapshots, I found gcc 4.2
fortifies its check for function pointer conversion and generates
abort for PEM_read_X509_AUX() and similar wrappers.

There was an old discussion about casting pointer to function
issue - "Why does casting a function generate a run-time abort?",
see http://gcc.gnu.org/ml/gcc/2004-06/msg01017.html

While understanding the reasoning of gcc develepers to catch
unportable code earlier, I want to rise question again in a hope
it would be useful to think of the whole issue again.

First of all, I did not find an answer why among many ways to shoot
yourself in a portability foot gcc chooses to fight so seriously
this particular case - function pointer conversions (it generates
runtime abort instead of function call). In a modern C world,
there are many portability issues triggered by undefined behavior
that can be detected at compile time. Is it feasible to generate
abort() for famous "a[i] = i++" code? For incompatible data pointer
conversion? For possible int types size mismatch? For non-portable
varargs manipulations?

Historically in C it was possible to workaround portability issues
in highly platform-dependent code with various #ifdef tricks -
selection of right snippet for target platform was in programmers
hand for well-written code. By generating runtime abort
unconditionally in a tricky case, we prevent a programmer from
using C as a "high level assembler" to write platform-dependent
code like threads/co-routines/message dispatcher libraries -
remember objective-c runtime.

Second, even with all that efforts put in guarding against
function pointer conversion, they failed to catch all cases.
Supposedly it is due to C language abstract machine philosophy -
it allows programmer to see target machine data and address
types. Unless C pointers goes far away from cpu-dependant
address, there should be impossible to catch all cases.
But that language will not be C language.

Consider the following code snippet, we see different
behavior from various gcc versions:

- cast_direct() generates abort since one of 3.x versions
- cast_via_void_fptr() still works with 4.1 but aborts with 4.2
- cast_via_assignment() works for 4.2 too
- cast_via_union() works and I suppose it should be a legally C
- cast_via_memcpy()... I did not bother to write it but you
see it is just another *legally* way to knowingly shoot compiler
check in a foot.

So it is still possible to generate call sequence with casted type despite
additional barriers from the 4.2 compiler.

Isn't it better to stop trying unstoppable ways of using C?
For example, in that particular openssl case introducing
runtime abort resulted in a following check-in from
openssl develepors:

"[13329]: Some C compilers produce warnings or compilation errors
if an attempt is made to directly cast a function of one type to what
it considers and incompatible type. In particular gcc 3.4.2.
Add new openssl_fcast macro to place functions into a form
where the compiler will allow them to be cast. "

Notice! You did not archive your goal - making programmers
writing portable code. Instead, they put a workaround against
your particular compiler, insisting on doing things in a way
they like.

With gcc 4.2 stricter check going in release, I foresee another
check-in, to cast via assign, union or other *workaround*.

To prevent that ridiculous compiler vs develepers war,
my proposition is:

At least revert back 4.2 conversion behavior to 4.1 behavior:
i.e. warn about direct function pointer conversion but
allow casts via void_fptr. That catches most illegal unintentional
code but allow people to use explisit type conversions for saying
compiler that they know that they do (for example, casting
const void* argument to void * agrument - that should
safe for all possible targets, I think).

At most, revert back even 3.4.2 introduced behavior and
do not generate illegal insn for warned suspicious conversion
but generate casted type function call as usual.
It is just impractical to make artificial barriers for
some undefined behavior while leaving others intact.
Abort for all undefined behavior - that is not a C way, too.
By the way, portability warnings should be  just warnings.

To summarize my opinion let me say that in guarding programmer
against undefined behavior current gcc made some steps in
a particular way. But the whole direction is wrong - when it will
be completed, we will have gcc be a compiler for other language then C.
Fortunately, for now that way is incomplete. ;)

---- here is a sample code ----

typedef void (*void_fptr) (void);

int foo (void * a)  { return 0; }

int cast_direct(void)
{
    return (((int (*) (const void *))  foo) (0));
}

int cast_via_void_fptr(void)
{
    return (((int (*) (const void *)) ((void_fptr) foo)) (0));
}

int cast_via_assignment(void)
{
    int (*bar) (const void *) = (int (*) (const void *))foo;
    return bar (0);
}

int cast_via_union(void)
{
    union {
        int (*foo) (void *);
        int (*bar) (const void *);
    } u;
    u.foo = foo;
    return u.bar (0);
}