gcc/cc1plus without executable bit, small, and "data" rather than "ELF 64-bit executable"

[james harvey <jamespharvey20 at gmail dot com>]

**In short**

I've come up with a build environment that a gcc bootstrap build fails
in.  This is because gcc/cc1plus, gcc/f951, or other executables don't
have executable bits set, are a bit smaller than in successful builds,
and the "file" command reports the files as being "data" rather than
"ELF 64-bit LSB executable..."

What could possibly cause this?  Are the executables like gcc/cc1plus
built incrementally in stages, and some stage is failing for me?


**With more details**

I'm tracking down a bug.  I think it's in the linux kernel 4
implementation of the overlay filesystem type, but it's possible it's
a gcc bug.  Let me start by creating 3 "lists" of commands

LIST 1
=====
sudo mkdir -p /sandbox/gcc
sudo mkdir -p /sandbox/hidden/gcc/{merged,workdir,dev.upperdir,dev.workdir,proc.upperdir,proc.workdir,sys.upperdir,sys.workdir}
sudo modprobe overlay
sudo mount -t overlay -o
lowerdir=/,upperdir=/sandbox/gcc,workdir=/sandbox/hidden/gcc/workdir
overlay /sandbox/hidden/gcc/merged
sudo mount -t overlay -o
lowerdir=/dev,upperdir=/sandbox/hidden/gcc/dev.upperdir,workdir=/sandbox/hidden/gcc/dev.workdir
overlay /sandbox/hidden/gcc/merged/dev
sudo mount -t overlay -o
lowerdir=/proc,upperdir=/sandbox/hidden/gcc/proc.upperdir,workdir=/sandbox/hidden/gcc/proc.workdir
overlay /sandbox/hidden/gcc/merged/proc
sudo mount -t overlay -o
lowerdir=/sys,upperdir=/sandbox/hidden/gcc/sys.upperdir,workdir=/sandbox/hidden/gcc/sys.workdir
overlay /sandbox/hidden/gcc/merged/sys
sudo chroot /sandbox/hidden/gcc/merged /bin/su - mdarling

LIST 2
=====
sudo dnf install wget zip tar texinfo-tex flex bison libmpc-devel isl
isl-devel ncurses-devel git gcc gcc-c++
git clone git://gcc.gnu.org/git/gcc.git gcc.git

LIST 3
=====
mkdir gcc.git.build
cd gcc.git.build
../gcc.git/configure --disable-multilib
make -j8

On Fedora 21 (kernel 3.19.7-200.fc21), if I execute LIST 1, 2, and 3,
sequentially, gcc builds perfectly.  (With an overlay filesystem
chroot wrapping up everything in LIST 2 and 3 inside of it.)

On Fedora 22 (kernel 4.0.4-303.fc22), if I do the same, gcc fails to
build with errors like:

xgcc: error trying to exec 'cc1plus': execvp: No such file or directory
Makefile:653: recipe for target 'array_type_info.lo' failed

Sometimes it fails complaining about other executables such as f951.

But, on Fedora 22 (kernel 4) I can execute the same commands omitting
"-j8" to avoid a parallel build, and gcc builds perfectly.

And, on Fedora 22 (kernel 4) I can use "-j8" to run a parallel build
if I skip mounting the overlay filesystem and chrooting into it.

So, on Fedora 22 (kernel 4), I can build gcc in parallel, or I can
build gcc serially in an overlay chroot, but I cannot do both at the
same time - building gcc in parallel in an overlay chroot.

I think the failure happens in stage 3.  It's at least 45 minutes in -
perhaps an hour or two.

The executable it fails on exists, and appears as:
-rw-rw-r--. 1 username username 190876345 Jun  7 03:15 cc1plus
And file cc1plus reports
"cc1plus: data"

In a separate build directory without running a parallel build, the
executable appears as:
-rwxrwxr-x. 1 username username 205126944 June  7 19:33 cc1plus
And file cc1plus reports
"cc1plus: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux),
dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for
GNU/Linux 2.6.32, not stripped

The above byte counts are on the same fresh installation, from a VM
snapshot.  So, the byte counts should be identical.

So, the "broken" version is a bit smaller than it should be, misses
the executable bit, and is data rather than an ELF 64-bit LSB
executable.