[Bug ipa/96040] [10/11 Regression] Compiled code causes SIGBUS at -O2

[josephcsible at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=96040

--- Comment #2 from Joseph C. Sible <josephcsible at gmail dot com> ---
Andrew Gierth posted this to the Lua mailing list:

> I think I see what's happening here, but I don't think I have an account
> on the gcc bug tracker to post it there (feel free to forward this).
> It's not (I think) a confusion over how many arguments the specialized
> tostringbuff.part.0.isra function has, but rather over the type, or
> equivalently the register allocation, for the first parameter.
> 
> The callsite is putting num->value_.n into %rdi, and &space into %rsi,
> as if the function were declared as taking (long long v, char *buf)
> rather than (double v, char *buf) which would require that num->value_.n
> be placed in %xmm0 and &space into %rdi.
> 
> But the code of the function itself is assuming that the incoming values
> are in %xmm0 and %rdi - %xmm0 is not touched because it's already in the
> right place for the snprintf call, while %rdi is left as the first arg
> to snprintf. The value 0x3ff0000000000000 is of course 1.0 as a double,
> which naturally does not work well as a pointer so it blows up.

I agree with the gist of that. I think this is more of a calling convention
issue than an argument count issue. It's as if the following pseudo-C were
written:

static int tostringbuff.part.0.isra.0 (union { long long i; double n; } value_
/* rdi */, char *str /* rsi */);

void addnum2buff (int *buff, struct TValue *num) {
    if(num->tt_ == 3) {
        buff += snprintf(space,50,"%lld",num->value_.i);
    } else {
        buff += tostringbuff.part.0.isra.0(num->value_ /* rdi */, space /* rsi
*/);
    }
}

static int tostringbuff.part.0.isra.0 (double n /* xmm0 */, char *str /* rdi
*/) {
    int len;
    len = snprintf(str,50,"%.14g",n);
    if(str[strspn(str, "-0123456789")] == '\0') {
      str[len++] = '.';
      str[len++] = '0';
    }
    return len;
}

In English, the compiler couldn't make up its mind as to how
tostringbuff.part.0.isra.0 was declared. At the call site, the first parameter
was a union, but at the definition, the first parameter was a double. Mixing up
a union and its element is fine when they only live in memory since they have
the same address, but here it was in a register. Since the calling convention
puts those types in different kinds of registers, what the function thought was
a pointer was actually the double value, predictably causing a crash when it
was dereferenced.