This is the mail archive of the
mailing list for the GCC project.
Re: gcc parallel make check
- From: Jakub Jelinek <jakub at redhat dot com>
- To: VandeVondele Joost <joost dot vandevondele at mat dot ethz dot ch>
- Cc: "gcc at gcc dot gnu dot org" <gcc at gcc dot gnu dot org>, "fortran at gcc dot gnu dot org" <fortran at gcc dot gnu dot org>
- Date: Wed, 3 Sep 2014 11:59:48 +0200
- Subject: Re: gcc parallel make check
- Authentication-results: sourceware.org; auth=none
- References: <908103EDB4893A42920B21D3568BFD93150E8FDF at MBX13 dot d dot ethz dot ch> <908103EDB4893A42920B21D3568BFD93150E8FFF at MBX13 dot d dot ethz dot ch> <20140903093001 dot GT17454 at tucnak dot redhat dot com> <908103EDB4893A42920B21D3568BFD93150EA47A at MBX13 dot d dot ethz dot ch>
- Reply-to: Jakub Jelinek <jakub at redhat dot com>
On Wed, Sep 03, 2014 at 09:37:19AM +0000, VandeVondele Joost wrote:
> > It is intentional. With -j it is essentially a fork bomb, just don't use it.
> well, silently ignoring it for just this target did cost me a lot of time, while an eventual fork bomb would have been dealt with much more quickly.
> >> Somewhat related.... is there a rule of thumb on how is the granularity of
> >> parallel check decided ? E.g. check-fortran seems to be limited to about
> >> ~5 parallel targets, which is few for a typical server (but of course a
> >> welcome speedup already).
> >The splitting has some cost (e.g. lots of various checks are cached, with
> >split jobs they need to be done in each separate goal), and the goal of the
> >split is toplevel make check parallelization, not individual directory or
> >language testing. For the latter perhaps more fine grained split could be
> >useful, but how would one find out if it is a toplevel make check, or say
> >make -C gcc check where you test many languages, or check-gfortran?
> the cost must be small compared to the possible gain... on a 32 core
> server, testing of fortran FE changes would be 4x larger. I notice that
It depends. For make -j2 if you split check-gfortran alone into 32 pieces,
check-gcc into 32 pieces, check-g++ into 32 pieces, libstdc++ into 32 pieces
etc., it might be too much.
The problem with too fine-grained split beyond some cost to start the
testing in the goal, and running various cached tests, is also that once you
want to split a single *.exp job into smaller parts, you need to use
wildcards, but then it is a maintainance problem, you don't want to test
anything more than once, or not at all, even if new tests with weirdo names
are added later.
> even on a full check, the Fortran tests are still running when the number
> of processes is already way below 32. However, the longest running (by a
> few minutes) are those:
> expect -- /usr/share/dejagnu/runtest.exp --tool gcc lto.exp weak.exp tls.exp ipa.exp tree-ssa.exp debug.exp dwarf2.exp fixed-point.exp vxworks.exp cilk-plus.exp vmx.exp pch.exp simulate-thread.exp x86_64-costmodel-vect.exp i386-costmodel-vect.exp spu-costmodel-vect.exp ppc-costmodel-vect.exp charset.exp noncompile.exp tsan.exp graphite.exp compat.exp
> expect -- /usr/share/dejagnu/runtest.exp --tool g++ lto.exp tls.exp gcov.exp debug.exp dwarf2.exp cilk-plus.exp pch.exp bprob.exp simulate-thread.exp vect.exp charset.exp tsan.exp graphite.exp compat.exp struct-layout-1.exp ubsan.exp tm.exp gomp.exp dfp.exp tree-prof.exp stackalign.exp plugin.exp guality.exp asan.exp ecos.exp
> so can those be run more independently ?
It is a moving target, new tests are added every day. I'm trying to adjust
it during stage3/stage4 occassionally, but it also very much depends on
which target it is (e.g. i?86/x86_64 has many more tests in i386.exp then
other targets in their gcc.target), how fast the compiler is on the target
(e.g. on some targets -g is much slower than on others, etc.).