[Bug c/47530] New: [trans-mem] tail call optimization problem with _ITM_commitTransaction

[patrick.marlier at gmail dot com]

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47530

           Summary: [trans-mem] tail call optimization problem with
                    _ITM_commitTransaction
           Product: gcc
           Version: trans-mem
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c
        AssignedTo: unassigned@gcc.gnu.org
        ReportedBy: patrick.marlier@gmail.com
                CC: rth@gcc.gnu.org, aldyh@gcc.gnu.org


According to:
http://gcc.gnu.org/ml/gcc/2011-01/msg00314.html
http://gcc.gnu.org/ml/gcc/2011-01/msg00279.html

The tail call optimization should not be possible on _ITM_commitTransaction
because this function can go (longjmp) to _ITM_beginTransaction and thus the
stack will be corrupted.

Here ASM:
_function:
        pushq   %r12
        pushq   %rbp
        pushq   %rbx
        subq    $16, %rsp
...
        call    _ITM_beginTransaction
.L8:
...
        call    _ITM_WU8
        addq    $16, %rsp
        popq    %rbx
        popq    %rbp
        popq    %r12
        jmp     _ITM_commitTransaction

In this example, if _ITM_commitTransaction rolls back to L8, the function
epilogue will restore wrong values to registers (modified by the function
_ITM_commitTransaction) when leaving the _function.

Moreover I think that the function _ITM_beginTransaction should have the flag
ECS_RETURNS_TWICE because it is how it behaves (ie the transaction can abort
and longjmp behind the _ITM_beginTransaction call).

Draft patch proposed in http://gcc.gnu.org/ml/gcc/2011-01/msg00279.html

Patrick Marlier.