Re: gcc parallel make check

[Jakub Jelinek <jakub at redhat dot com>]

On Wed, Sep 03, 2014 at 09:37:19AM +0000, VandeVondele  Joost wrote:
> > It is intentional.  With -j it is essentially a fork bomb, just don't use it.
> 
> well, silently ignoring it for just this target did cost me a lot of time, while an eventual fork bomb would have been dealt with much more quickly.
> 
> >> Somewhat related.... is there a rule of thumb on how is the granularity of
> >> parallel check decided ?  E.g.  check-fortran seems to be limited to about
> >> ~5 parallel targets, which is few for a typical server (but of course a
> >> welcome speedup already).
> >
> >The splitting has some cost (e.g. lots of various checks are cached, with
> >split jobs they need to be done in each separate goal), and the goal of the
> >split is toplevel make check parallelization, not individual directory or
> >language testing.  For the latter perhaps more fine grained split could be
> >useful, but how would one find out if it is a toplevel make check, or say
> >make -C gcc check where you test many languages, or check-gfortran?
> 
> the cost must be small compared to the possible gain... on a 32 core
> server, testing of fortran FE changes would be 4x larger.  I notice that

It depends.  For make -j2 if you split check-gfortran alone into 32 pieces,
check-gcc into 32 pieces, check-g++ into 32 pieces, libstdc++ into 32 pieces
etc., it might be too much.
The problem with too fine-grained split beyond some cost to start the
testing in the goal, and running various cached tests, is also that once you
want to split a single *.exp job into smaller parts, you need to use
wildcards, but then it is a maintainance problem, you don't want to test
anything more than once, or not at all, even if new tests with weirdo names
are added later.

> even on a full check, the Fortran tests are still running when the number
> of processes is already way below 32.  However, the longest running (by a
> few minutes) are those:
> 
> expect -- /usr/share/dejagnu/runtest.exp --tool gcc lto.exp weak.exp tls.exp ipa.exp tree-ssa.exp debug.exp dwarf2.exp fixed-point.exp vxworks.exp cilk-plus.exp vmx.exp pch.exp simulate-thread.exp x86_64-costmodel-vect.exp i386-costmodel-vect.exp spu-costmodel-vect.exp ppc-costmodel-vect.exp charset.exp noncompile.exp tsan.exp graphite.exp compat.exp
> expect -- /usr/share/dejagnu/runtest.exp --tool g++ lto.exp tls.exp gcov.exp debug.exp dwarf2.exp cilk-plus.exp pch.exp bprob.exp simulate-thread.exp vect.exp charset.exp tsan.exp graphite.exp compat.exp struct-layout-1.exp ubsan.exp tm.exp gomp.exp dfp.exp tree-prof.exp stackalign.exp plugin.exp guality.exp asan.exp ecos.exp
> 
> so can those be run more independently ?

It is a moving target, new tests are added every day.  I'm trying to adjust
it during stage3/stage4 occassionally, but it also very much depends on
which target it is (e.g. i?86/x86_64 has many more tests in i386.exp then
other targets in their gcc.target), how fast the compiler is on the target
(e.g. on some targets -g is much slower than on others, etc.).

	Jakub