This is the mail archive of the
gcc-patches@gcc.gnu.org
mailing list for the GCC project.
Re: [PATCH] gcc parallel make check
- From: Jakub Jelinek <jakub at redhat dot com>
- To: David Malcolm <dmalcolm at redhat dot com>
- Cc: Mike Stump <mikestump at comcast dot net>, VandeVondele Joost <joost dot vandevondele at mat dot ethz dot ch>, "gcc at gcc dot gnu dot org" <gcc at gcc dot gnu dot org>, "fortran at gcc dot gnu dot org" <fortran at gcc dot gnu dot org>, "gcc-patches at gcc dot gnu dot org" <gcc-patches at gcc dot gnu dot org>
- Date: Thu, 11 Sep 2014 10:06:40 +0200
- Subject: Re: [PATCH] gcc parallel make check
- Authentication-results: sourceware.org; auth=none
- References: <20140905143740 dot GL17454 at tucnak dot redhat dot com> <908103EDB4893A42920B21D3568BFD93150F414C at MBX23 dot d dot ethz dot ch> <20140905145304 dot GM17454 at tucnak dot redhat dot com> <908103EDB4893A42920B21D3568BFD93150F7F45 at MBX23 dot d dot ethz dot ch> <908103EDB4893A42920B21D3568BFD93150F816B at MBX23 dot d dot ethz dot ch> <229476F6-B901-4C6E-AE0B-3A53521AE996 at comcast dot net> <1410381512 dot 28338 dot 9 dot camel at surprise> <20140910210822 dot GK17454 at tucnak dot redhat dot com> <20140910212334 dot GL17454 at tucnak dot redhat dot com> <20140911075123 dot GN17454 at tucnak dot redhat dot com>
- Reply-to: Jakub Jelinek <jakub at redhat dot com>
On Thu, Sep 11, 2014 at 09:51:23AM +0200, Jakub Jelinek wrote:
> I can't find how to query the -jN value passed to make check by the user
> though, both $(MFLAGS) and $(MAKEFLAGS) only contain something like
> --jobserver-fds=3,5 -j from which it is not possible to find out how many
> goals would be the upper reasonable limit. Running too many goals would
> waste time (once scheduled, the goal would only wildcard all the test, and
> for all of them find in the *-parallel directory the test has been run
> already), running too few could prevent good parallelization.
After a little googling, it seems there is no way to do that :(, unless
one e.g. attempts to find the command line of the topmost parent make
and scan it through ps or something.
There is an option to touch say *-parallel/finished file once any of the
check-parallel-gcc-{1,2,...} goals is done (because when it finishes, it
means all the tests for the particular check-$lang that are parallelizable
have either finished, or at least touched their file) and not start runtest
at all if finished already exists, but guess it would be still undesirable to have
tens of thousands of goals by default, so perhaps we could go with say
128 subgoals by default and have some env var to override it, so on the
really highly parallel boxes you'd specify
make -j512 -k check GCC_TEST_PARALLEL_SLOTS=512
or similar.
Jakub