comparing parallel test runs

[Aldy Hernandez <aldyh at redhat dot com>]

Hi folks.

I've been having troubles comparing the results of different test runs 
for quite some time, and have finally decided to whine about it. 
Perhaps someone can point out to whatever I may be doing wrong.

I generally do "make check -k -j60" on two different trees and compare 
the results with:

cd /dir/current-build-tree
/gcc-sources/contrib/compare_tests /dir/previous-build-tree .

However, this gives false positives virtually every time:

New tests that FAIL:

g++.dg/tree-prof/pr79259.C execution,    -g
gcc.dg/tree-prof/ic-misattribution-1.c execution,    -g
gcc.dg/tree-prof/merge_block.c execution,    -g
gcc.dg/tree-prof/section-attr-1.c execution,    -g
gcc.dg/tree-prof/section-attr-3.c execution,    -g
...
...

New tests that PASS:

g++.dg/tree-prof/partition1.C compilation,  -g
g++.dg/tree-prof/partition2.C compilation,  -g
g++.dg/tree-prof/partition3.C compilation,  -g
g++.dg/tree-prof/pr79259.C compilation,  -g
...
...

Old tests that passed, that have disappeared: (Eeek!)

g++.dg/tree-prof/indir-call-prof-2.C compilation,  -g
g++.dg/tree-prof/morefunc.C compilation,  -g
g++.dg/tree-prof/pr63581.C compilation,  -g
gcc.dg/tree-prof/20050826-2.c compilation,  -g
gcc.dg/tree-prof/bb-reorg.c compilation,  -g
...
...

etc
etc
etc

More often than not the tests that get mentioned are in the 
gcc.dg/tree-prof directory, but I've seen failures in gcc.dg/tree-ssa, 
gcc.dg/vect and others.

When I run these tests manually (make check-gcc 
RUNTESTFLAGS=tree-ssa.exp=whatever.c), they succeed.

Running with smaller -jNN usually yields less false positives, but 
rarely do I get a clean run.

This has been happening for as long as I can remember (which may just be 
12 months ;-)).

Has anyone seen this behavior?  Is it my test box?  Are there known 
problems with parallel checks?

FWIW, I'm running on an x86-64 Fedora 25 running a 4.9.6-200.fc25.x86_64 
kernel.  The machine has 56 cores, and has dejagnu 1.6 installed.

Thanks.
Aldy

p.s. Attached is the full output from contrib/compare_tests.

Attachment: est
Description: Text document