[Bug optimization/11366] New: [3.3 regression] miscompiles XDM-AUTHORIZATION-1 key generation and/or validation in XFree86 at -O2

["debian-gcc at lists dot debian dot org" <gcc-bugzilla at gcc dot gnu dot org>]

PLEASE REPLY TO gcc-bugzilla@gcc.gnu.org ONLY, *NOT* gcc-bugs@gcc.gnu.org.

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=11366

           Summary: [3.3 regression] miscompiles XDM-AUTHORIZATION-1 key
                    generation and/or validation in XFree86 at -O2
           Product: gcc
           Version: 3.3
            Status: UNCONFIRMED
          Severity: critical
          Priority: P2
         Component: optimization
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: debian-gcc at lists dot debian dot org
                CC: gcc-bugs at gcc dot gnu dot org
 GCC build triplet: i386-linux / powerpc-linux
  GCC host triplet: i386-linux / powerpc-linux
GCC target triplet: i386-linux / powerpc-linux

This is not a complete bug report. Any hints and help to track down this report
are welcome.

[forwarded from http://bugs.debian.org/196090]

Rechecked with 3.3 CVS 20030627, with HEAD xfree86 does finish the build due to
some ICE. Setting the optimization to -O2 -fno-strict-aliasing doesn't help
either. Compiling with -O1 doesn't show the error. Seen at least on i386-linux
and powerpc-linux.

The symptom can be found in /var/log/xdm.log:
AUDIT: Fri Jun 27 20:28:48 2003: 18641 X: client 2 rejected from local host
  Auth name: XDM-AUTHORIZATION-1 ID: -1
Xlib: connection to ":0.0" refused by server
Xlib: Invalid XDM-AUTHORIZATION-1 key (failed key comparison)
/usr/X11R6/bin/xrdb: Can't open display ':0'


Bug submitter writes:

When testing XFree86 4.2.1-7 on my PowerPC I found that compiling the
tree with -O2 (per Debian Policy) and GCC 3.3 caused the X server to
break such that XDM-AUTHORIZATION-1 Xauth tokens could not be validated.

I have not narrowed this down.  I do know that using the 4.2.1-6 server
with the 4.2.1-7 libraries worked fine, so I suspect a miscompilation on
the server side, but it is conceivable that the problem is really on the
client/library side and that GCC 3.3 is exposing a very subtle error in
the code.  Or something like that.

Compiling with -O instead of -O2 made the problem go away, and that's
the optimization level that was used for the 4.2.1-7 packages for
PowerPC which were uploaded.

Anyway, I have heard rumors that the GCC team knows that there are
problems with this optimization level on PowerPC in GCC 3.3.  Something
about a bunch of new subtargets being added to the ppc/ELF backend in
this release?

Please nag me for more information if too much time passes and I have
not followed up.

Alternatively some person might be willing to track this down for me
using the recipe above.  Just take the 4.2.1-7 source package and remove
the special case optimization stepdown for powerpc from the debian/rules
file.

Compile XFree86, install the resulting packages, and use xdm to
(attempt) to start a session.  You'll also notice that the bustage keeps
the xdm greeter from working properly.


Jack Howard adds:
   Franz Sirl, the ppclinux devtool maintainer, has kindly
built the current rawhide XFree86 4.3.0 srpms against 
gcc-3.3 from 6/18/03 on Yellow Dog Linux. He is unable to
reproduce the authentication problems we see on debian
under either xdm or kdm. Perhaps we build XFree86 differently
enough from RedHat that only we tickled the gcc bug. However
I did find out that the gcc 3.3 that RedHat uses is not
a patched version of the gcc 3.3 branch but rather a special
rh branch of gcc-3.3. One wonders if they would see the
problem if they used the stock gcc-3.3 branch instead and
if there are any fixes trapped in the gcc-3.3 rh branch that
haven't been put into the main gcc-3.3 branch.