This is the mail archive of the
mailing list for the GCC project.
Re: [RFC, Fortran] Avoid race on testsuite temporary files
- From: Janne Blomqvist <blomqvist dot janne at gmail dot com>
- To: Yvan Roux <yvan dot roux at linaro dot org>
- Cc: gcc mailing list <gcc at gcc dot gnu dot org>, Maxim Kuvyrkov <maxim dot kuvyrkov at linaro dot org>, Mikael Morin <mikael dot morin at sfr dot fr>
- Date: Fri, 11 Dec 2015 13:12:37 +0200
- Subject: Re: [RFC, Fortran] Avoid race on testsuite temporary files
- Authentication-results: sourceware.org; auth=none
- References: <CAD57uCcYgc7d2eixqe++zBYqPAEe=eQy6KW5uaG+XbyezHx8ew at mail dot gmail dot com>
On Wed, Dec 9, 2015 at 4:27 PM, Yvan Roux <email@example.com> wrote:
> as it was raised in
> https://gcc.gnu.org/ml/gcc-patches/2015-10/msg01540.html we experiment
> random failures in gfortran validation when it is run in parallel (-j
> 8). The issues occurs because of concurrent access to the same file,
> the first two patches fixed some of them by changing the file names
> used, but there are still remaining conflicts (6 usages of foo, 8 of
> test.dat). This is easy to fix and I've a patch for that, but there is
> another issue and I'd like to have your thoughts on it.
> There is a little bit more than 1000 testcases which use IO without
> explicit file names, ~150 use scratches (a temporary file named
> gfortrantmp + 6 extra char is created) and the others, which only
> specify the io unit number, use a file named fort.NN with NN the
> number of the io unit. We see conflicts on these generated files, as
> lot of testcases use the same number, the most used are:
> 10 => 150
> 99 => 70
> 6 => 31
> 11 => 27
> 1 => 23
> I started to change the testcases to use scratches when it is possible
> and before finding that there is that many to fix, and I also had
> conflicts on the generated scratch names. The options I see to fix
> that are:
> 1- Move all these testcases into an IO subdir and change the testsuite
> to avoid parallelism in that directory.
> 2- Use scratches when possible and improve libgfortran file name
> generation, I don't know well fortran but is it possible to change the
> file name patterns for scratches and io unit files ?
For the "io unit files", I don't think we can change it. fort.NN is a
common convention, and I suspect a lot of people rely on it. As for
scratch files, I'm surprised you're seeing conflicts. At least on
Linux/POSIX targets it should use mk(o)stmp(), which should make sure
a unique file name is chosen and opened with exclusive access
(O_CREAT|O_EXCL). Furthermore, on POSIX targets it unlinks the file
immediately after opening it, further reducing the possibility of any
conflict. So what target are you compiling for?
> 3- Change the io unit numbers used, as it was suggested on irc, but I
> find it a bit painful to maintain.
Is it really that much more work than 1)? If you don't want to keep
track of a count, just generate a random number between 10 and 2^31,
and use that as the unit number? Should at least make conflicts rare.
> Any comments are welcome.