comparing parallel test runs

[Aldy Hernandez]

Hi folks.

I've been having troubles comparing the results of different test runs 
for quite some time, and have finally decided to whine about it. 
Perhaps someone can point out to whatever I may be doing wrong.

I generally do "make check -k -j60" on two different trees and compare 
the results with:

cd /dir/current-build-tree
/gcc-sources/contrib/compare_tests /dir/previous-build-tree .

However, this gives false positives virtually every time:

New tests that FAIL:

g++.dg/tree-prof/pr79259.C execution,    -g
gcc.dg/tree-prof/ic-misattribution-1.c execution,    -g
gcc.dg/tree-prof/merge_block.c execution,    -g
gcc.dg/tree-prof/section-attr-1.c execution,    -g
gcc.dg/tree-prof/section-attr-3.c execution,    -g
...
...

New tests that PASS:

g++.dg/tree-prof/partition1.C compilation,  -g
g++.dg/tree-prof/partition2.C compilation,  -g
g++.dg/tree-prof/partition3.C compilation,  -g
g++.dg/tree-prof/pr79259.C compilation,  -g
...
...

Old tests that passed, that have disappeared: (Eeek!)

g++.dg/tree-prof/indir-call-prof-2.C compilation,  -g
g++.dg/tree-prof/morefunc.C compilation,  -g
g++.dg/tree-prof/pr63581.C compilation,  -g
gcc.dg/tree-prof/20050826-2.c compilation,  -g
gcc.dg/tree-prof/bb-reorg.c compilation,  -g
...
...

etc
etc
etc

More often than not the tests that get mentioned are in the 
gcc.dg/tree-prof directory, but I've seen failures in gcc.dg/tree-ssa, 
gcc.dg/vect and others.

When I run these tests manually (make check-gcc 
RUNTESTFLAGS=tree-ssa.exp=whatever.c), they succeed.

Running with smaller -jNN usually yields less false positives, but 
rarely do I get a clean run.

This has been happening for as long as I can remember (which may just be 
12 months ;-)).

Has anyone seen this behavior?  Is it my test box?  Are there known 
problems with parallel checks?

FWIW, I'm running on an x86-64 Fedora 25 running a 4.9.6-200.fc25.x86_64 
kernel.  The machine has 56 cores, and has dejagnu 1.6 installed.

Thanks.
Aldy

p.s. Attached is the full output from contrib/compare_tests.
-------------- next part --------------
# Comparing directories
## Dir1=/home/cygnus/aldyh/bld/pristine/: 8 sum files
## Dir2=.: 8 sum files

# Comparing 8 common sum files
## /bin/sh /home/cygnus/aldyh/src/gcc/contrib/compare_tests  /tmp/gxx-sum1.39318 /tmp/gxx-sum2.39318
New tests that FAIL:

g++.dg/tree-prof/pr79259.C execution,    -g
gcc.dg/tree-prof/ic-misattribution-1.c execution,    -g
gcc.dg/tree-prof/merge_block.c execution,    -g
gcc.dg/tree-prof/section-attr-1.c execution,    -g
gcc.dg/tree-prof/section-attr-3.c execution,    -g
gcc.dg/tree-prof/tracer-1.c execution,    -g
gcc.dg/tree-prof/val-prof-8.c execution,    -g

New tests that PASS:

g++.dg/tree-prof/partition1.C compilation,  -g
g++.dg/tree-prof/partition2.C compilation,  -g
g++.dg/tree-prof/partition3.C compilation,  -g
g++.dg/tree-prof/pr79259.C compilation,  -g
gcc.dg/tree-prof/comp-goto-1.c compilation,  -g
gcc.dg/tree-prof/ic-misattribution-1.c compilation,  -g
gcc.dg/tree-prof/merge_block.c compilation,  -g
gcc.dg/tree-prof/pr47187.c compilation,  -g
gcc.dg/tree-prof/pr52150.c compilation,  -g
gcc.dg/tree-prof/section-attr-1.c compilation,  -g
gcc.dg/tree-prof/stringop-1.c compilation,  -g
gcc.dg/tree-prof/tracer-1.c compilation,  -g
gcc.dg/tree-prof/tracer-1.c scan-tree-dump-times tracer "c =" 2
gcc.dg/tree-prof/update-cunroll-2.c compilation,  -g
gcc.dg/tree-prof/val-prof-8.c compilation,  -g
gcc.dg/tree-prof/wcoverage-mismatch.c compilation,  -g
gcc.dg/tree-ssa/20030807-6.c scan-tree-dump-times vrp1 "ABS_EXPR " 0
gcc.dg/tree-ssa/pr18589-6.c scan-tree-dump-times optimized " \\* " 4

Old tests that passed, that have disappeared: (Eeek!)

g++.dg/tree-prof/indir-call-prof-2.C compilation,  -g
g++.dg/tree-prof/morefunc.C compilation,  -g
g++.dg/tree-prof/pr63581.C compilation,  -g
gcc.dg/tree-prof/20050826-2.c compilation,  -g
gcc.dg/tree-prof/bb-reorg.c compilation,  -g
gcc.dg/tree-prof/cold_partition_label.c compilation,  -g
gcc.dg/tree-prof/ic-misattribution-1a.c compilation,  -g
gcc.dg/tree-prof/inliner-1.c compilation,  -g
gcc.dg/tree-prof/peel-1.c compilation,  -g
gcc.dg/tree-prof/pr49299-1.c compilation,  -g
gcc.dg/tree-prof/pr49299-2.c compilation,  -g
gcc.dg/tree-prof/pr52027.c compilation,  -g
gcc.dg/tree-prof/pr79587.c compilation,  -g
gcc.dg/tree-prof/prof-robust-1.c compilation,  -g
gcc.dg/tree-prof/section-attr-2.c compilation,  -g
gcc.dg/tree-prof/stringop-2.c compilation,  -g
gcc.dg/tree-prof/time-profiler-1.c compilation,  -g
gcc.dg/tree-prof/time-profiler-3.c compilation,  -g
gcc.dg/tree-prof/unroll-1.c compilation,  -g
gcc.dg/tree-prof/val-prof-6.c compilation,  -g
gcc.dg/tree-prof/val-prof-7.c compilation,  -g
gcc.dg/tree-prof/val-prof-9.c compilation,  -g
gcc.dg/tree-ssa/20030530-2.c scan-tree-dump-times dom2 "\\+" 1
gcc.dg/tree-ssa/20030530-2.c scan-tree-dump-times dom2 "first_gp_reg_save" 1
gcc.dg/tree-ssa/20030730-1.c scan-tree-dump-times dom2 "if " 0
gfortran.dg/vect/vect-1.f90   -O   scan-tree-dump-times vect "vectorized 3 loops" 1

Old tests that failed, that have disappeared: (Eeek!)

g++.dg/tree-prof/inline_mismatch_args.C execution,    -g
g++.dg/tree-prof/pr63581.C execution,    -g
gcc.dg/tree-prof/bb-reorg.c execution,    -g
gcc.dg/tree-prof/cold_partition_label.c execution,    -g
gcc.dg/tree-prof/peel-1.c execution,    -g
gcc.dg/tree-prof/pr34999.c execution,    -g
gcc.dg/tree-prof/pr49299-1.c execution,    -g
gcc.dg/tree-prof/pr50907.c execution,    -g
gcc.dg/tree-prof/pr52027.c execution,    -g
gcc.dg/tree-prof/pr79587.c execution,    -g
gcc.dg/tree-prof/time-profiler-1.c execution,    -g
gcc.dg/tree-prof/time-profiler-3.c execution,    -g
gcc.dg/tree-prof/val-prof-6.c execution,    -g
gcc.dg/tree-prof/val-prof-7.c execution,    -g
gcc.dg/tree-prof/val-profiler-threads-1.c execution,    -g

## Differences found: 
# 1 differences in 8 common sum files found