[Bug c++/81943] New: Wrong ABI in class methods returning structs for the Windows amd64 target

["blaffablaffa at gmail dot com" <gcc-bugzilla at gcc dot gnu dot org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81943

            Bug ID: 81943
           Summary: Wrong ABI in class methods returning structs for the
                    Windows amd64 target
           Product: gcc
           Version: 7.1.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c++
          Assignee: unassigned at gcc dot gnu.org
          Reporter: blaffablaffa at gmail dot com
  Target Milestone: ---

test case:


        struct retstruct{
                int a;
        };

        struct badABI{
                retstruct f(int arg){
                        retstruct ret;
                        ret.a = arg;
                        arg++;
                        return ret;
                }
        };

        int main(){
                badABI test;
                auto testret = test.f(234);
        };


g++ assembly output (x86_64-w64-mingw32-g++ -S -std=c++11 -o main.S main.cpp)
around the badABI::f call:


        leaq    -1(%rbp), %rax
        movl    $234, %edx      #2nd arg, int
        movq    %rax, %rcx      #1st arg, this pointer
        call    _ZN6badABI1fEi
        movl    %eax, -8(%rbp)  #returned struct in eax


cl.exe (version 19, Visual Studio 2017) output:


        mov      r8d, 234   ; 000000eaH         #3rd argument, int
        lea      rdx, QWORD PTR testret$[rsp]   #2nd argument, return buffer
        lea      rcx, QWORD PTR test$[rsp]      #1st argument, this pointer
        call     badABI::f


Note that this is not fixed by -fpcc-struct-return, because the this pointer
and the hidden return struct buffer pointer are inverted:


        leaq    -8(%rbp), %rax
        leaq    -1(%rbp), %rdx  #2nd argument, this pointer
        movl    $234, %r8d      #3rd argument, int
        movq    %rax, %rcx      #1st argument, return buffer
        call    _ZN6badABI1fEi


As far as I know this last convention isn't used anywhere on windows, so it's
completely wrong. On the other hand, the first convention appears to follow the
specifications for small struct returns
(https://msdn.microsoft.com/en-us/library/7572ztz4(v=vs.120).aspx), but I can't
explain why the latest cl.exe does not appear to use it.



There are projects where this discrepancy has been taken into account, the one
I know of is https://github.com/SteamRE/open-steamworks : it is a reverse
engineering of the Steamworks SDK (started when it wasn't public), and they
have a number of instances (see for example
https://github.com/SteamRE/open-steamworks/blob/master/Open%20Steamworks/ISteamUser017.h#L44
macro STEAMWORKS_STRUCT_RETURN_0) where the return of structs is fixed by
actually returning void and passing a reference to the return buffer.



clang appears to emit the right assembly when using -cc1 -triplet
x86_64-pc-windows-msvc18.0.0 (default of clang-cl):


        leaq    48(%rsp), %rcx  #1st argument, this pointer
        leaq    40(%rsp), %rdx  #2nd argument, return buffer
        movl    $234, %r8d      #3rd argument, int
        callq   "?f@badABI@@QEAA?AUretstruct@@H@Z"


excerpt of badABI::f:


        movq    %rdx, %rax      #return buffer pointer in rax (correct as per
microsoft ABI)
        movl    %r8d, 12(%rsp)
        movq    %rcx, (%rsp)
        movl    12(%rsp), %r8d
        movl    %r8d, (%rdx)
        movl    12(%rsp), %r8d
        addl    $1, %r8d
        movl    %r8d, 12(%rsp)