c/4426: gcc-3.0 incorrectly compiles OpenSSL 0.9.6

[alaric at babcom dot com]

>Number:         4426
>Category:       c
>Synopsis:       gcc-3.0 produces subtly non-functional libssl
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    unassigned
>State:          open
>Class:          wrong-code
>Submitter-Id:   net
>Arrival-Date:   Sun Sep 30 11:26:01 PDT 2001
>Closed-Date:
>Last-Modified:
>Originator:     Phil Stracchino
>Release:        3.0
>Organization:
Babylon Communications
>Environment:
System: Linux babylon5 2.4.6-ac2 #2 Sat Jul 7 18:23:45 PDT 2001 i586 unknown
Architecture: i586
host: i586-pc-linux-gnu
build: i586-pc-linux-gnu
target: i586-pc-linux-gnu
configured with: ../gcc-3.0.0/configure --prefix=/usr
>Description:

OpenSSL cannot be correctly compiled on Linux with gcc3.  The configure
and compile process will appear completely normal, the resulting libraries
will pass all tests, and working SSL certificates can be produced with the
openssl binary.  However, OpenSSH compiled against the resulting libraries
will expose that a problem exists, problem, as all attempts to unlock
passphrase-protected SSH2 keys will fail at the PEM_read_PrivateKey call.  
(PEM_read_PrivateKey is actually a macro wrapper around PEM_ASN1_read.)
The failure occurs within libssl, and does not produce any error either
from libssl or in the calling OpenSSH program; the attempt to decrypt the
key simply fails as though an incorrect password was supplied.  If OpenSSL
is recompiled with gcc-2.91.66 or gcc-2.95.3, OpenSSH compiled against the
resulting library will work perfectly, regardless of whether OpenSSH was
compiled using gcc-2.91.66, gcc-2.95.3 or gcc-3.0.

The flawed libssl produced by gcc-3.0 also causes Apache processes 
compiled with mod_ssl support to fail, although in this case the proximal 
mechanism of the failure is unknown.

The incorrect code generation is obviously something very subtle, as all 
other libssl functions appear to work correctly and the library passes all 
formal tests.  Only actual functional testing exposes the problem.

>How-To-Repeat:

Compile OpenSSL-0.9.6{a,b} on a Linux platform using gcc-3.0.  Compile 
OpenSSH-2.9* against the resulting libssl using any gcc version.  Create 
a password-protected rsa or dsa SSH2 key.  Start ssh-agent, then run 
ssh-add .ssh/id_{d,r}sa.  Observe that the correct password fails to 
unlock the key.  Trace the execution of ssh-add with gdb and observe that 
the call to PEM_read_PrivateKey in authfile.c always returns NULL even for 
the correct password.

>Fix:

No known fix using gcc3.  As far as I can tell, gcc-3.0 simply will not
produce a properly working libssl regardless of compilation options.
>Release-Note:
>Audit-Trail:
>Unformatted: