c/3771: Component of openssl fails with optimisation greater than -O1

[dave at datatone dot co dot uk]

>Number:         3771
>Category:       c
>Synopsis:       Component of openssl fails with optimisation greater than -O1
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    unassigned
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Mon Jul 23 01:56:01 PDT 2001
>Closed-Date:
>Last-Modified:
>Originator:     dave@datatone.co.uk
>Release:        unknown-1.0
>Organization:
>Environment:

>Description:
The public domain software 'openssl-0.9.6b' fails when compiled with gcc-3.0 and gcc-3.0-20010615 when -O3 or -O2 optimisation is used. It works with -O0 and -O1.
The symptom is that:
openssl rsa -noout -text -in <key-file>
produces an error to the effect that the key file could not be read.
I discovered that this was due to the fact that some pointer arithmetic in the file:
openssl-0.9.6b/crypto/pem/pem_lib.c
was not working. A pointer to char is passed to the function: PEM_get_EVP_CIPHER_INFO. This pointer points to a buffer containing the header from the key file. the intention is to check various fields in the header. There is code thus:

header+=11;
if(*header=='4') ...

I found that header was being incremented by 11*2 characters, rather than by 11. I also found that the problem would come and go as I placed 'print' statements into the code to try to discover what the problem was. Because of this apparent dependence on the local code, I began to suspect problems introduced in optimisation. I therefore tried compiling the package with different levels of optimisation turned on. I found that it worked with -O0 and -O1; but not with -O2 and -O3.

>How-To-Repeat:
Attempt to compile and run openssl-0.9.6b
>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted: