[Bug lto/77997] New: Miscompilation due to LTO on aarch64

["yyc1992 at gmail dot com" <gcc-bugzilla at gcc dot gnu dot org>]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=77997

            Bug ID: 77997
           Summary: Miscompilation due to LTO on aarch64
           Product: gcc
           Version: 6.2.1
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: lto
          Assignee: unassigned at gcc dot gnu.org
          Reporter: yyc1992 at gmail dot com
  Target Milestone: ---

I'm seeing a miscompilation of LLVM's tablegen on AArch64 by gcc 6.2.1 when LTO
is enabled. I've tried very hard to reduce it but unfortunately it wasn't very
successful this time and the current repro is still 8000 lines of code.

Attached are the source and resulting binaries. (Edit: the tarball is too big
(3M) so I uploaded to gist instead. Please find it here
https://gist.githubusercontent.com/yuyichao/6c24d4a4bc374425906138359a44479c/raw/f5edb6ae8205d5e4d1eb044443a7fb900f15711f/report.md)

The `code/` directory has a simple cmake projects reduced from the LLVM one (I
can turn that into a makefile or a shell script on request but the current form
should be pretty simple already). To reproduce make a `build/` directory in
`code/` and run `CFLAGS='-flto -O3' CXXFLAGS='-flto -O3' LDFLAGS='-O3 -flto'
cmake .. -DCMAKE_BUILD_TYPE=Release; make llvm-tblgen; bin/llvm-tblgen`. Remove
the `-flto` should get rid of the error in the last command.

Changes in seemingly unrelated lines can also make the error go away. (If
there's anything I learnt from reducing it, the error seems to appear only when
the code is complex....). One of such changes is commenting out
`SCTrans.PredTerm = Preds;` close to the end of `CodeGenSchedule.cpp` (used to
generate the `good/` version included). In fact, removing almost any lines in
this file can make the error go away even though not a single line of code
there should be executed.

The `bad/` and the `good/` directories conatins compilation results using the
flags mentioned above with the unmodified code and the code with the one line
commented out. They have all the object files, binary files and the disassemble
of the resulting executable/the bad function. The asm's are disassembled from
the final binary since I don't know how to get it directly when compiling with
LTO.

The direct error seems to be in `CodeGenRegister::computeSubRegs` in the branch
before the `printf("5\n")`. The `DenseMap::insert` method (which is called
twice in this function and nowhere else) is inlined but returns corrupted
iterator sometimes when the inserted key already exists in the map causing the
check to fail. The difference of asm of this function is in the toplevel of the
tarball.

Original repro is compiling LLVM with LTO on on AArch64. The compilation should
fail when generating target information for AArch64.

GCC version is GCC binary package from ArchLinux ARM repo. `gcc --version`
gives `gcc (GCC) 6.2.1 20160830`