[Bug tree-optimization/62032] FAIL: vsnprintf-chk.c execution, -O2 -flto -fno-use-linker-plugin -flto-partition=none

[amker.cheng at gmail dot com]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=62032

--- Comment #3 from bin.cheng <amker.cheng at gmail dot com> ---
I did some investigation and think it's a latent bug of lto which reveaded by
r213585.

Before the revision, pass_fold_builtins::execute calls gimple_fold_builtins
directly to fold __builtin___vsnprintf_chk (_26, _25, 0, 4294967295, "%d - %c",
ap);.  It works fine by simplifying the builtin into __builtin_vsnprintf (_26,
_25, "%d - %c", ap);.

After the revision, it calls to
fold_stmt->gimple_fold_call->gimple_fold_builtin to do the job.  But before
that, it calls gimple_call_builtin_p to verify that it IS a valid builtin call,
which in turn calls gimple_builtin_call_types_compatible_p to check that types
between parameter and argument match each other.  Well, the check fails on the
last parameter/argument, as dumped below:

TREE_TYPE of ARGUMENT:
 <record_type 0x7fc34cc9fc78 va_list SI
    size <integer_cst 0x7fc34ce71dc8 type <integer_type 0x7fc34ce7c0a8
bitsizetype> constant 32>
    unit size <integer_cst 0x7fc34ce71de0 type <integer_type 0x7fc34ce7c000
sizetype> constant 4>
    align 32 symtab 0 alias set -1 canonical type 0x7fc34ce899d8
    fields <field_decl 0x7fc34ce87130 __ap
        type <pointer_type 0x7fc34ce84000 type <void_type 0x7fc34ce7cf18 void>
            public unsigned SI size <integer_cst 0x7fc34ce71dc8 32> unit size
<integer_cst 0x7fc34ce71de0 4>
            align 32 symtab 0 alias set 3 canonical type 0x7fc34ce84000
            pointer_to_this <pointer_type 0x7fc34ce8a348>>
        unsigned SI file <built-in> line 0 col 0 size <integer_cst
0x7fc34ce71dc8 32> unit size <integer_cst 0x7fc34ce71de0 4>
        align 32 offset_align 64
        offset <integer_cst 0x7fc34ce71df8 constant 0>
        bit offset <integer_cst 0x7fc34ce71e58 constant 0> context <record_type
0x7fc34ce899d8 __va_list>> context <translation_unit_decl 0x7fc34df310f0
D.4035>
    pointer_to_this <pointer_type 0x7fc34ccaa2a0> chain <type_decl
0x7fc34ce87da8 __va_list>>

TREE_TYPE of PARAMETER of function BUILT_IN_VSNPRINTF_CHK:
 <reference_type 0x7fc34ce89c78
    type <record_type 0x7fc34ce899d8 __va_list SI
        size <integer_cst 0x7fc34ce71dc8 constant 32>
        unit size <integer_cst 0x7fc34ce71de0 constant 4>
        align 32 symtab 0 alias set 7 canonical type 0x7fc34ce899d8
        fields <field_decl 0x7fc34ce87130 __ap type <pointer_type
0x7fc34ce84000>
            unsigned SI file <built-in> line 0 col 0 size <integer_cst
0x7fc34ce71dc8 32> unit size <integer_cst 0x7fc34ce71de0 4>
            align 32 offset_align 64
            offset <integer_cst 0x7fc34ce71df8 constant 0>
            bit offset <integer_cst 0x7fc34ce71e58 constant 0> context
<record_type 0x7fc34ce899d8 __va_list>>
        pointer_to_this <pointer_type 0x7fc34ccc4888> reference_to_this
<reference_type 0x7fc34ccaa0a8> chain <type_decl 0x7fc34ce87098 __va_list>>
    unsigned SI size <integer_cst 0x7fc34ce71dc8 32> unit size <integer_cst
0x7fc34ce71de0 4>
    align 32 symtab 0 alias set -1 canonical type 0x7fc34ce89c78>


The root cause lies in gcc/lto/lto-lang.c:lto_init, there is below code
snippet:

  if (TREE_CODE (va_list_type_node) == ARRAY_TYPE)
    {
      tree x = build_pointer_type (TREE_TYPE (va_list_type_node));
      lto_define_builtins (x, x);
    }
  else
    {
      lto_define_builtins (va_list_type_node,
               build_reference_type (va_list_type_node));
    }

While prototype of lto_define_builtins is like below.
static void
lto_define_builtins (tree va_list_ref_type_node ATTRIBUTE_UNUSED,
             tree va_list_arg_type_node ATTRIBUTE_UNUSED);

Apparently, the arguments passed to lto_define_builtins are mis-matched.  In
fact, I think it should be consistent with function calls to c_define_builtins
in c front-end.

I think above analysis also explains why it happens only with lto.

I will send a patch fixing this soon.

Thanks,
bin