Re: comparing parallel test runs

[Richard Biener <richard dot guenther at gmail dot com>]

On Wed, May 17, 2017 at 12:23 PM, Aldy Hernandez <aldyh@redhat.com> wrote:
> Hi folks.
>
> I've been having troubles comparing the results of different test runs for
> quite some time, and have finally decided to whine about it. Perhaps someone
> can point out to whatever I may be doing wrong.
>
> I generally do "make check -k -j60" on two different trees and compare the
> results with:
>
> cd /dir/current-build-tree
> /gcc-sources/contrib/compare_tests /dir/previous-build-tree .
>
> However, this gives false positives virtually every time:
>
> New tests that FAIL:
>
> g++.dg/tree-prof/pr79259.C execution,    -g
> gcc.dg/tree-prof/ic-misattribution-1.c execution,    -g
> gcc.dg/tree-prof/merge_block.c execution,    -g
> gcc.dg/tree-prof/section-attr-1.c execution,    -g
> gcc.dg/tree-prof/section-attr-3.c execution,    -g
> ...
> ...
>
> New tests that PASS:
>
> g++.dg/tree-prof/partition1.C compilation,  -g
> g++.dg/tree-prof/partition2.C compilation,  -g
> g++.dg/tree-prof/partition3.C compilation,  -g
> g++.dg/tree-prof/pr79259.C compilation,  -g
> ...
> ...
>
> Old tests that passed, that have disappeared: (Eeek!)
>
> g++.dg/tree-prof/indir-call-prof-2.C compilation,  -g
> g++.dg/tree-prof/morefunc.C compilation,  -g
> g++.dg/tree-prof/pr63581.C compilation,  -g
> gcc.dg/tree-prof/20050826-2.c compilation,  -g
> gcc.dg/tree-prof/bb-reorg.c compilation,  -g
> ...
> ...
>
> etc
> etc
> etc
>
> More often than not the tests that get mentioned are in the gcc.dg/tree-prof
> directory, but I've seen failures in gcc.dg/tree-ssa, gcc.dg/vect and
> others.
>
> When I run these tests manually (make check-gcc
> RUNTESTFLAGS=tree-ssa.exp=whatever.c), they succeed.
>
> Running with smaller -jNN usually yields less false positives, but rarely do
> I get a clean run.
>
> This has been happening for as long as I can remember (which may just be 12
> months ;-)).
>
> Has anyone seen this behavior?  Is it my test box?  Are there known problems
> with parallel checks?
>
> FWIW, I'm running on an x86-64 Fedora 25 running a 4.9.6-200.fc25.x86_64
> kernel.  The machine has 56 cores, and has dejagnu 1.6 installed.

Likely auto-fdo, see Andis recent patch to imrpove the siutation a
bit.  It helped
for me (but I'm only using -j6 on the only machine I see the issue).

Richard.

> Thanks.
> Aldy
>
> p.s. Attached is the full output from contrib/compare_tests.