GCC 8, ppc64-le and problems with -fstack-protector-strong

[Jeffrey Walton]

Hi Everyone,

We are tracking an issue on ppc64-le and GCC 8. We can duplicate it on
GCC112 when using /opt/cfarm/gcc8-r257824 compiler. Also see
https://github.com/weidai11/cryptopp/issues/588.

We have the issue isolated, but its not really reduced yet. When '-O2
-fstack-protector-strong' is used during a compile then we have some
failed self tests. When '-O3 -fstack-protector-strong' is used during
a compile then everything is OK. (It is not backwards; -O3 is OK, and
-O2 is bad).

The source code is clean under the sanitizers on the platform. The
problem does not occur under GCC 4.8.5 (default compiler) and GCC 7
(installed at /opt/cfarm/gcc-latest).

The source file that is causing the problems under Power8 is
https://github.com/weidai11/cryptopp/blob/master/rijndael-simd.cpp#L536
. Its an awful file and nearly everything is inlined.

I've tried looking at the disassemblies but I don't have the knowledge
to spot the trouble. I'm not sure how to troubleshoot things further.

Suggestions on the next step would be very helpful.

Thanks in advance.