Infinite recursion due to builtin pattern detection

["Paulo Matos" <pmatos at broadcom dot com>]

Hi,

I have noticed an interesting behaviour based on the memset builtin test on 4.8.1. If you compile memset-lib.i with -O3 you get an infinite recursion.
$ cat memset-lib.i
# 1 "/home/pmatos/work/fp_gcc-25x/gcc/testsuite/gcc.c-torture/execute/builtins/memset-lib.c"
# 1 "<command-line>"
# 1 "/home/pmatos/work/fp_gcc-25x/gcc/testsuite/gcc.c-torture/execute/builtins/memset-lib.c"
# 1 "/home/pmatos/work/fp_gcc-25x/gcc/testsuite/gcc.c-torture/execute/builtins/lib/memset.c" 1
extern void abort (void);
extern int inside_main;

__attribute__ ((__noinline__))
void *
memset (void *dst, int c, long unsigned int n)
{
  while (n-- != 0)
    n[(char *) dst] = c;





  if (inside_main && n < 2)
    abort ();


  return dst;
}
# 1 "/home/pmatos/work/fp_gcc-25x/gcc/testsuite/gcc.c-torture/execute/builtins/memset-lib.c" 2

$ ~/work/tmp/gcc_4.8.1-build/gcc/cc1 -O3 memset-lib.i -o-
        .file   "memset-lib.i"
 memset
Analyzing compilation unit
Performing interprocedural optimizations
 <*free_lang_data> <visibility> <early_local_cleanups> <*free_inline_summary> <whole-program> <profile_estimate> <cp> <inline> <pure-const> <static-var>Assembling functions:
 memset .text
        .p2align 4,,15
        .globl  memset
        .type   memset, @function
memset:
.LFB0:
        .cfi_startproc
        testq   %rdx, %rdx
        je      .L6
        subq    $8, %rsp
        .cfi_def_cfa_offset 16
        movsbl  %sil, %esi
        call    memset
        addq    $8, %rsp
        .cfi_def_cfa_offset 8
        ret
        .p2align 4,,10
        .p2align 3
.L6:
        movq    %rdi, %rax
        ret
        .cfi_endproc
.LFE0:
        .size   memset, .-memset
        .ident  "GCC: (GNU) 4.8.1"
        .section        .note.GNU-stack,"",@progbits



So, it seems the GCC is smart enough to detect the pattern for memset but once it replaces the loop into the function a recursion is created because the function its replacing the loop in is called memset itself.
Interestingly GCC doesn't actually fail the test, probably due to interactions with other files and functions but this compilation unit does generate incorrect output so I think there might be a bug lurking here.

Another thing I cannot grasp is that the test for inside_main completely disappears. We do know n <= 2 after the loop but we have no information for inside_main so I would assume we still need to test inside_main to know if we should abort.

What do you think?

Paulo Matos