Re: gcc-2.95.2 breaks stack alignment from aligned main()

[<stevenj at gil-galad dot mit dot edu>]

Dear gcc developers,

My colleague, Dr. Matteo Frigo (athena at fftw dot org) has found the
source of the stack alignment bug I reported in gcc:

	http://egcs.cygnus.com/ml/gcc-bugs/1999-11/msg00259.html

We have no patch to correct the problem, but I hope that the analysis I
describe below will (a) help convince the gcc developers that this is
indeed a bug and (b) help people more familiar with the gcc source code to
correct the problem.

The problem, apparently, is that gcc's stack alignment algorithm does not
correctly handle the case of nested subroutine calls f(g(x)).

Matteo changed my test program to the simpler:

int one(int count)
{
     double foobar;
     if ((((long) &foobar) & 0x7)) {
	  printf("one %d\n", count);
	  abort();
     }
     return 1;
}

void blah(int foo)
{
     double foobar;
     if ((((long) &foobar) & 0x7)) {
	  printf("blah\n");
	  abort();
     }
}

int main(void)
{
     one(1);
     blah(one(2));
     return 0;
}

The output (after compiling with gcc --save-temps) is:
	one 2
	Aborted (core dumped)

The source of the misalignment is described below by analysing the
assembly language output.  Note that main *is* aligned; I checked by
adding a similar alignment check into main() and got the same result
(misalignment in one(2)).  The main() check was removed from the above
code in order to make the resulting assembly code cleaner.

main:
	pushl %ebp
	movl %esp,%ebp
	subl $8,%esp
	addl $-12,%esp
	pushl $1
        # ******* assume stack has alignment A (mod 8) here
	call one
        # ******* then the alignment is A (mod 8) here too
	addl $16,%esp
	addl $-12,%esp
	addl $-12,%esp
        # ******* and also here (after adding a multiple of 8 to %esp)
	pushl $2
        # ******* hence the stack alignment is (A + 4) (mod 8) here
	# ******* after pushing 4 bytes...misaligned if A was aligned (0)!
	call one
	addl $16,%esp
	movl %eax,%eax
	pushl %eax
	call blah

So, even if the stack is initially aligned (A = 0), it will be misaligned
upon the second call to one().

Matteo looked at the code generation algorithm in gcc, and he described
the problem as follows.  For a function call f(a, b) gcc seems to do:

	 adjust stack pointer
	 push b
	 push a
	 call f

i.e., it adjusts the stack before pushing arguments.  The algorithm
fails when compiling f(a(b)), because it produces:

        adjust sp for f
        adjust sp for a, without accounting for previous step
	push b
	call a
	push return value from a which is in some register
	call f

[ He thinks the bug is somewhere in calls.c, but had trouble finding it. ]

I hope this is helpful, and that someone will be able to find a patch.

Cordially,
Steven G. Johnson