Power 8 in-core crypto not working as expected

[Jeffrey Walton]

Hi Everyone,

I'm on gcc rather than gcc-help because we need to talk with some GCC
devs who can help take this further.

I have implementation for AES on Power 8 using GCC's built-ins. Its
available for inspection and download at
https://github.com/noloader/AES-Power8. The problem is, it does not
arrive at the correct results on GCC112 (ppc64-le) or GCC119 (AIX, big
endian).

The source file is the reduced, minimal test case. It uses
pre-caclulated subkeys so we've removed that variable from the
equation. It also uses the null vector (string of 0's) as the message,
so that variable has been removed from the equation too.

About all we are left with is loading a subkey, calling vcipher to
perform a single round of encryption, and assigning the result back to
a variable. Lather, rinse, repeat.

For the crypto side of things I've consulted with Andy Polyakov of the
OpenSSL project. I believe we are doing everything we should be as far
as the crypto goes, including the subkey byte-swaps on LE machines.
Our subkey table is exactly the same as the one OpenSSL arrives at on
both LE and BE machines.

Would someone familiar with the processor and knowledge of GCC
built-in's please take a look at things. Suggestions for our next
steps would be greatly appreciated.

Thanks in advance,

Jeffrey Walton

==========

Here are the compiler versions.

  - GCC112 (Linux, little endian)

    $ gcc --version
    gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11)

  - GCC119 (AIX, big endian):

    $ gcc --version
    gcc (GCC) 6.1.0