target/3050: g++ snapshot 20010526 (x86) generates code which accesses below %esp

[jseward at acm dot org]

>Number:         3050
>Category:       target
>Synopsis:       g++ snapshot 20010526 (x86) generates code which accesses below %esp
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    unassigned
>State:          open
>Class:          wrong-code
>Submitter-Id:   net
>Arrival-Date:   Mon Jun 04 17:06:01 PDT 2001
>Closed-Date:
>Last-Modified:
>Originator:     Julian Seward
>Release:        gcc version 3.0 20010526 (prerelease)
>Organization:
>Environment:
Vanilla x86-RedHat 7.1 box
>Description:
When compiled with g++ shown below, with -O2 -mcpu=i686,
the resulting code for BandMatrix::ReSize(int n, int lb, int ub) reads memory below %esp towards the end of the function, which is potentially fatal if the stack is trashed by a signal delivery at that precise moment.
C++ programs thusly afflicted could occasionally
crash in a random, hard-to-reproduce manner.

This is very sensitive to flags.  The problem does not appear for any of the following flags:

   -O2 -mcpu=i586
   -O
   -O -mcpu=i686
   -O -mcpu=i586

I can only reproduce it with -O2 -mcpu=i686.

The code generated is actually correct in other respects.
g++'s error is to have retreated %esp prematurely, hence
leaving live data on the stack exposed to trashing by 
signal handlers for a very short period.  The relevant fragment of assembly is

.LCFI5:
        call    _ZN13GeneralMatrix6ReSizeEiii
        movl    %esi, (%esp)
        call    _ZNK10BandMatrix11CornerClearEv
.LEHE0:
        addl    $16, %esp

	# THESE TWO INSNS CONSTITUTE THE ERROR -- THEY
	# SHOULD BE THE OTHER WAY AROUND
        leal    -8(%ebp), %esp
        movl    4(%ebx), %eax

        popl    %ebx
        movl    %eax, last
        popl    %esi
        popl    %ebp
        ret

Single-stepping with gdb, immediately prior to insn
        movl    4(%ebx), %eax
we have
   (gdb) p/x $esp
   $1 = 0xbffff930
   (gdb) p/x (4 + $ebx)
   $2 = 0xbffff92c
ie  
   4 + %ebx  <  %esp
which does not seem good to me.  (it is a violation
of the user-space code's contract with the kernel).
>How-To-Repeat:
Compile with -O2 -mcpu=i686 -S.
>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted:
----gnatsweb-attachment----
Content-Type: text/x-c++src; name="bogon.ii"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="bogon.ii"

IyAyICJib2dvbi5jcHAiCmNsYXNzIFRyYWNlcgp7CiAgIGNvbnN0IGNoYXIqIGVudHJ5OwogICBU
cmFjZXIqIHByZXZpb3VzOwpwdWJsaWM6CiAgIFRyYWNlcihjb25zdCBjaGFyKik7CiAgIH5UcmFj
ZXIoKTsKICAgdm9pZCBSZU5hbWUoY29uc3QgY2hhciopOwogICBzdGF0aWMgdm9pZCBQcmludFRy
YWNlKCk7CiAgIHN0YXRpYyB2b2lkIEFkZFRyYWNlKCk7Cn07CgpzdGF0aWMgVHJhY2VyKiBsYXN0
OwoKaW5saW5lIFRyYWNlcjo6VHJhY2VyKGNvbnN0IGNoYXIqIGUpCiAgIDogZW50cnkoZSksIHBy
ZXZpb3VzKGxhc3QpIHsgbGFzdCA9IHRoaXM7IH0KCmlubGluZSBUcmFjZXI6On5UcmFjZXIoKSB7
IGxhc3QgPSBwcmV2aW91czsgfQoKaW5saW5lIHZvaWQgVHJhY2VyOjpSZU5hbWUoY29uc3QgY2hh
ciogZSkgeyBlbnRyeT1lOyB9CgoKY2xhc3MgR2VuZXJhbE1hdHJpeCB7CiAgIGludCB3dXJibGU7
CiBwdWJsaWM6CiAgIHZvaWQgUmVTaXplKGludCxpbnQsaW50KTsKICAgdm9pZCBDb3JuZXJDbGVh
cigpIGNvbnN0OwogICBHZW5lcmFsTWF0cml4KCk7Cn07CgpjbGFzcyBCYW5kTWF0cml4IDogcHVi
bGljIEdlbmVyYWxNYXRyaXggewogcHVibGljOgogICB2b2lkIFJlU2l6ZShpbnQsaW50LGludCk7
CiAgIHZvaWQgQ29ybmVyQ2xlYXIoKSBjb25zdDsKfTsKCgp2b2lkIEJhbmRNYXRyaXg6OlJlU2l6
ZShpbnQgbiwgaW50IGxiLCBpbnQgdWIpCnsKICAgVHJhY2VyIHRyKCJCYW5kTWF0cml4OjpSZVNp
emUiKTsKICAgaW50IGxvd2VyID0gKGxiPD1uKSA/IGxiIDogbi0xOwogICBpbnQgdXBwZXIgPSAo
dWI8PW4pID8gdWIgOiBuLTE7CiAgIEdlbmVyYWxNYXRyaXg6OlJlU2l6ZShuLG4sbioobG93ZXIr
MSt1cHBlcikpOwogICBDb3JuZXJDbGVhcigpOwp9Cgp2b2lkIEJhbmRNYXRyaXg6OkNvcm5lckNs
ZWFyKCkgY29uc3QKewp9Cgp2b2lkIEdlbmVyYWxNYXRyaXg6OlJlU2l6ZShpbnQgbiwgaW50IGxi
LCBpbnQgdWIpCnsKfQoKR2VuZXJhbE1hdHJpeDo6R2VuZXJhbE1hdHJpeCgpIDogd3VyYmxlKDQy
KQp7Cn0KCgppbnQgbWFpbiAoIHZvaWQgKQp7CiAgIEJhbmRNYXRyaXggenp6OwogICB6enouUmVT
aXplKDAsMCwwKTsKICAgcmV0dXJuIDA7Cn0K