[PATCH] Deduce automatically number of cores for -flto option.

[Martin Liška]

On 7/31/19 1:32 AM, Jakub Jelinek wrote:
> This broke a lot of tests.

Whoops.

> The logs show
> spawn -ignore SIGHUP /home/jakub/src/gcc/obj31/gcc/xgcc -B/home/jakub/src/gcc/obj31/gcc/ c_lto_pr83954_0.o c_lto_pr83954_1.o -fno-diagnostics-show-
> caret -fno-diagnostics-show-line-numbers -fdiagnostics-color=never -O2 -flto -flto-partition=1to1 -o gcc-dg-lto-pr83954-31.exe
> make[4]: *** write jobserver: Bad file descriptor.  Stop.
> make[4]: *** Waiting for unfinished jobs....
> make[4]: *** write jobserver: Bad file descriptor.  Stop.
> lto-wrapper: fatal error: make returned 2 exit status
> compilation terminated.
> collect2: fatal error: lto-wrapper returned 1 exit status
> compilation terminated.
> compiler exited with status 1
> FAIL: gcc.dg/lto/pr83954 c_lto_pr83954_0.o-c_lto_pr83954_1.o link, -O2 -flto -flto-partition=1to1 
> and similar, e.g. for x86_64-linux it was following regressions, on
> i686-linux also some tests in libgomp etc.
> Is -flto now really using all available CPUs for each compilation?

Yes, I can confirm that linking happens in N processed for each LTO test now.
It's caused by fact that current Dejagnu machinery does not pass a make jobserver
to gcc command invocations. On the other hand, our LTO tests are so tiny that we
should have always very low number of partitions.

>  Without
> jobserver that would like a fork-bomb, say if I have a CPU with 32 threads
> and do make check -j32, does that mean there are 1024 lto1s?
> Judging from http://gcc.gnu.org/ml/gcc-testresults/2019-07/msg03610.html
> I'm not alone.

One possible solution will be to adjust lto.exp:

	  set LTO_OPTIONS [list	\
	      {-O0 -flto -flto-partition=none -fuse-linker-plugin} \
	      {-O2 -flto -flto-partition=none -fuse-linker-plugin -fno-fat-lto-objects } \
	      {-O0 -flto -flto-partition=1to1 -fno-use-linker-plugin } \
	      {-O2 -flto -flto-partition=1to1 -fno-use-linker-plugin } \

and replace all -flto with -flto=1. But still, many individual tests set -flto by themselves.
Another solution would be to disable the auto-detection with an environment variable:

diff --git a/gcc/lto-wrapper.c b/gcc/lto-wrapper.c
index 353187c6043..bb6fb2b53ff 100644
--- a/gcc/lto-wrapper.c
+++ b/gcc/lto-wrapper.c
@@ -1423,7 +1423,7 @@ run_gcc (unsigned argc, char *argv[])
       auto_parallel = 0;
       parallel = 0;
     }
-  else if (!jobserver && parallel)
+  else if (!jobserver && parallel && !getenv ("LTO_NO_AUTO_PARALLEL"))
     {
       /* If there's no explicit usage of jobserver and
 	 parallel is enabled, then automatically detect
diff --git a/gcc/testsuite/lib/lto.exp b/gcc/testsuite/lib/lto.exp
index 25c934731df..e303894e0b0 100644
--- a/gcc/testsuite/lib/lto.exp
+++ b/gcc/testsuite/lib/lto.exp
@@ -209,6 +209,8 @@ proc lto_init { args } {
 	  ]
 	}
     }
+
+    setenv LTO_NO_AUTO_PARALLEL 1
 }
 
 #

Can you Jakub test the patch or the s/-flto/-flto=1 solutions please?
Thanks,
Martin