In https://github.com/NixOS/nixpkgs/issues/229470 Arnout Engelen noticed that 'make install -j' sometimes installs incomplete '$prefix/share/info/dir' file. This happens because 'make install-info' when running in parallel is executing equivalent of: if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1; then \ install-info --dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir /<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/gcc.info; \ if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1; then \ install-info --dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir /<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/gccinstall.info; \ if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1; then \ install-info --dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir /<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/cppinternals.info; \ if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1; then \ install-info --dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir /<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/cpp.info; \ if /<<NIX>>/bash-5.2-p15/bin/bash -c 'install-info --version' >/dev/null 2>&1; then \ install-info --dir-file=/<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/dir /<<NIX>>/x86_64-unknown-linux-musl-stage-static-gcc-14.0.0/share/info/gccint.info; \ On it's own 'install-info' does not lock $prefix/share/info/dir file atomically. As a result multiple parallel executions of 'install-info' compete and occasionally throw away work of one another. Right now `nixpkgs` works it around as https://github.com/NixOS/nixpkgs/pull/229898: --- gcc-12.2.0/gcc/Makefile.in 2022-08-19 10:09:52.280658631 +0200 +++ gcc-12.2.0-new/gcc/Makefile.in 2023-05-04 14:35:44.401420184 +0200 @@ -3781,6 +3781,11 @@ fi; \ fi +# We don't care about the order in which the info files are built, but +# install-info doesn't support multiple parallel invocations writing to +# the same `dir-file`, so we have to disable parallelism for that reason: +.NOTPARALLEL: install-info + # Install the info files. # $(INSTALL_DATA) might be a relative pathname, so we can't cd into srcdir # to do the install. It might not be enough to handle language-specific 'install-info' targets. What would be the best way to handle this race condition?
make all of 'install' not parallel?
(In reply to Sergei Trofimovich from comment #0) > --- gcc-12.2.0/gcc/Makefile.in 2022-08-19 10:09:52.280658631 +0200 > +++ gcc-12.2.0-new/gcc/Makefile.in 2023-05-04 14:35:44.401420184 +0200 > @@ -3781,6 +3781,11 @@ > fi; \ > fi > > +# We don't care about the order in which the info files are built, but > +# install-info doesn't support multiple parallel invocations writing to > +# the same `dir-file`, so we have to disable parallelism for that reason: > +.NOTPARALLEL: install-info Prerequisites of .NOTPARALLEL are ignored, so doesn't this un-parallelize building the entire gcc sub-dir?
(In reply to Jonathan Wakely from comment #2) > (In reply to Sergei Trofimovich from comment #0) > > --- gcc-12.2.0/gcc/Makefile.in 2022-08-19 10:09:52.280658631 +0200 > > +++ gcc-12.2.0-new/gcc/Makefile.in 2023-05-04 14:35:44.401420184 +0200 > > @@ -3781,6 +3781,11 @@ > > fi; \ > > fi > > > > +# We don't care about the order in which the info files are built, but > > +# install-info doesn't support multiple parallel invocations writing to > > +# the same `dir-file`, so we have to disable parallelism for that reason: > > +.NOTPARALLEL: install-info > > Prerequisites of .NOTPARALLEL are ignored, so doesn't this un-parallelize > building the entire gcc sub-dir? When I tested on small example '.NOTPARALLEL: target' serialized exactly prerequisites of 'target' and nothing more. 'info make' seems to confirm the behaviour: """ '.NOTPARALLEL' If '.NOTPARALLEL' is mentioned as a target with no prerequisites, all targets in this invocation of 'make' will be run serially, even if the '-j' option is given. Any recursively invoked 'make' command will still run recipes in parallel (unless its makefile also contains this target). If '.NOTPARALLEL' has targets as prerequisites, then all the prerequisites of those targets will be run serially. This implicitly adds a '.WAIT' between each prerequisite of the listed targets. *Note Disabling Parallel Execution: Parallel Disable. """ Here is the test that confirms it: $ cat Makefile all: a b a: 1 2 echo a 1: sleep 1 2: sleep 1 b: 3 4 echo b 3: sleep 1 4: sleep 1 .NOTPARALLEL: a $ make -j | LANG=C ts May 18 20:34:23 sleep 1 May 18 20:34:23 sleep 1 May 18 20:34:23 sleep 1 May 18 20:34:24 sleep 1 May 18 20:34:24 echo b May 18 20:34:24 b May 18 20:34:25 echo a May 18 20:34:25 a Note how it takes 'b' 1 second to finish (due to parallelism) while 'a' takes 2 seconds (targets are sequential).
That seems to be new in Make 4.4, because the manual for Make 4.3 says: '.NOTPARALLEL' If '.NOTPARALLEL' is mentioned as a target, then this invocation of 'make' will be run serially, even if the '-j' option is given. Any recursively invoked 'make' command will still run recipes in parallel (unless its makefile also contains this target). Any prerequisites on this target are ignored.
But perhaps make install could $(MAKE) NOT_PARALLEL=1 non-parallel-install and the makefile could have ifeq ($(NOT_PARALLEL),1) .NOTPARALLEL: endif ?