[Bug sanitizer/81715] asan-stack=1 redzone allocation is too inflexible

[arnd at linaro dot org]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=81715

--- Comment #10 from Arnd Bergmann <arnd at linaro dot org> ---
As far as I can tell, gcc doesn't merge stack slots that came from inline
functions, as in comment 1, or this example:

void baz (int *, int *, int *, int *, int *, int *);
static inline void foo (int a, int b, int c, int d, int e, int f)
{
    baz (&a, &b, &c, &d, &e, &f);
}
void
bar (int a, int b, int c, int d, int e, int f)
{
    foo (a, b, c, d, e, f);
    foo (a, b, c, d, e, f);
    foo (a, b, c, d, e, f);
    foo (a, b, c, d, e, f);
    foo (a, b, c, d, e, f);
    foo (a, b, c, d, e, f);
    foo (a, b, c, d, e, f);
    foo (a, b, c, d, e, f);
}

The frame sizes I see here are

gcc-8 -O2:                                            192 bytes
gcc-8 -O2 -fsanitize=address:                        3120 bytes
gcc-8 -O2 -fsanitize=kernel-address:                  192 bytes
gcc-8 -O2 -fsanitize=address asan-stack=0:            192 bytes
gcc-8 -O2 -fsanitize=kernel-address asan-stack=1:    3120 bytes
clang -O2:                                             72 bytes
clang -O2 -fsanitize=address:                          88 bytes
clang -O2 -fsanitize=kernel-address:                  888 bytes
clang -O2 -fsanitize=address asan-stack=0:            104 bytes
clang -O2 -fsanitize=kernel-address asan-stack=0:     104 bytes

(note: clang -fsanitize=kernel-address defaults to asan-stack=1, while gcc
defaults to asan-stack=0. gcc-5 and gcc-8 have identical output).