parallel testsuite results differ from a single task approach

[Tim Prince]

On 12/20/2012 11:06 PM, Dennis Clarke wrote:
> I gad received some good advice to try using gmake -j X -k check for some X > 1 when running a testsuite in order to speed the process up. So I decided to test that theory with an eye on the results. I was surprised to see that the results when running "gmake -j 8 -k check" differ from a simple "gmake -k check" in the area of go and libmudflap :
>
>      1 ) parallel testsuite :
>
>          http://gcc.gnu.org/ml/gcc-testresults/2012-12/msg01903.html
>
>      2 ) single process testsuite :
>
>          http://gcc.gnu.org/ml/gcc-testresults/2012-12/msg01935.html
>
>
> In parallel we see this for "go" :
>
>                  === go tests ===
>
>
> Running target unix
> FAIL: go.test/test/stack.go execution,  -O2 -g
>
>                  === go Summary ===
>
> # of expected passes            3293
> # of unexpected failures        1
> # of expected failures          4
> # of untested testcases         4
> /home/dclarke/pgm/build/gcc-4.7.2_2.6.32-279.14.1.el6.x86_64.004/gcc/testsuite/go/../../gccgo  version 4.7.2 (Blastwave.org Inc. time_rfc-3339=2012-12-20 11:23:25+00:00)
>
>
>
>
> While the single process apporach reports :
>
>
>                  === go Summary ===
>
> # of expected passes            3294
> # of expected failures          4
> # of untested testcases         4
> /home/dclarke/pgm/build/gcc-4.7.2_2.6.32-279.14.1.el6.x86_64.004/gcc/testsuite/go/../../gccgo  version 4.7.2 (Blastwave.org Inc. time_rfc-3339=2012-12-20 11:23:25+00:00)
>
>
> Perhaps the go test "go.test/test/stack.go" is somehow sensitive to running on a busy machine? I am only guessing.
>
> The "libmudflap tests" results are wildly different.
>
> So at first glance it appears that the testsuite will in fact product different results if one chooses to use "gmake -j X" for X>1 but I am not sure why.  Felt is was worth pointing out as maybe no one else has done the comparison.
>
> Dennis
>
>
Did you read any reports from recent years about problems with this?  If 
a test dupiicates itself while accessing the same file system, it may 
create a kind of race condition.
If the additional failures are time-outs, it's to be expected on a "busy 
machine."  In particular, you may have multiple tests contending for a 
single rotating disk.  Number of cores, OS, and other factors will make 
a large difference.  How are we to know what you aren't telling?

-- 
Tim Prince