Running the testsuite in parallel

[Marc Glisse]

On Fri, 20 Apr 2012, Ian Lance Taylor wrote:

> Marc Glisse <marc.glisse@inria.fr> writes:
>
>> building gcc in parallel (make -j): works ok
>>
>> installing gcc in parallel: not necessary (used to fail regularly, but
>> I haven't tried recently)
>>
>> running the testsuite in parallel: not so parallel
>>
>> On an 8-core PC, for a c,c++ build, make -k -j check leaves the
>> computer at least 60% idle (I looked at top periodically), and for a
>> huge part of the time it uses at most 2 cores (and often only 1).
>>
>> Since it looks like the testsuite is supposed to run in parallel, I
>> wonder if I am doing it the wrong way, if there are options to pass or
>> a different command to run?
>
> The horrible test harness that we use, DejaGNU, does not support running
> tests in parallel.  Running them in parallel is implemented as a hack in
> which the tests are manually split up into sets, and the various sets
> are passed to DejaGNU in parallel.

Yes, I saw your blog on the subject.

> So what you are seeing is that
> manually splitting up tests does not lead to optimal use of a parallel
> build.
>
> There is no simple fix, but as the actual splitting was done several
> years ago it is possible that new tests have overloaded some of the
> subsets.  Perhaps simply resplitting the tests would lead to better
> parallelism.  I don't know.  If you want to investigate, look at
> check_gcc_parallelize and friends in gcc/Makefile.in.

If the only problem was unbalanced targets, I should see a high load at 
the beginning that becomes lower and lower as only a few long-running 
targets remain. Here, the load does go down, but even at the beginning it 
is low, which I find suspicious.

If I figure something out, I'll post it, but it seems hard...

Thanks for your answer,

-- 
Marc Glisse