This is the mail archive of the mailing list for the GCC project.

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: RFC [testsuite] Obey --load-average

On 08/03/2017 11:45 AM, Jeff Law wrote:
> On 08/02/2017 11:34 PM, Daniel Santos wrote:
> So does this perform better than make -j X -l X?  I use that with good
> success.
> jeff

Sorry for my slow response!

For a short answer, if you have 8 CPU cores and you run make -j8 -l8
check then everything is fine until cron needs to do something that eats
4 CPUs and you end up with a load average of 12.  This is because the
make jobs of the test harness are very long running and make's only
control lever for regulating load is to not start new jobs until the
load falls.  So if all jobs are already launched, make has no way of
reducing the load until all tests on that test set have completed.

Each job that make launches runs an expect script that iterates through
the tests for that .exp file and performs its own parallelization using
lock files written to the directory specified by the
GCC_RUNTEST_PARALLELIZE_DIR environment variable.  It's not a very
pretty picture, but it is a functioning integration.  What this patch
does is to interject a load test and sleep within the expect script's
loop prior to a job grabbing a new lock file.

I'm sure that there are more elegant solutions, but they are likely more
labor intensive.  There are still a few challenges with this approach
because parallelization isn't implemented in a uniform way throughout
the testsuite, so I'm working on disentangling it from other code in
gcc/testsuite/lib/gcc-defs.exp so that libstdc++-v3 can use the same

This will be helpful for me when I need to update a bootstrap and then
re-run tests on both a baseline and a patched branch since I can run two
instances of the testsuite and not end up over-loaded.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]