Re: [RFC, Fortran] Avoid race on testsuite temporary files

[Yvan Roux <yvan dot roux at linaro dot org>]

Hi Janne,

On 11 December 2015 at 12:12, Janne Blomqvist <blomqvist.janne@gmail.com> wrote:
> On Wed, Dec 9, 2015 at 4:27 PM, Yvan Roux <yvan.roux@linaro.org> wrote:
>> Hi,
>>
>> as it was raised in
>> https://gcc.gnu.org/ml/gcc-patches/2015-10/msg01540.html we experiment
>> random failures in gfortran validation when it is run in parallel (-j
>> 8).  The issues occurs because of concurrent access to the same file,
>> the first two patches fixed some of them by changing the file names
>> used, but there are still remaining conflicts (6 usages of foo, 8 of
>> test.dat). This is easy to fix and I've a patch for that, but there is
>> another issue and I'd like to have your thoughts on it.
>>
>> There is a little bit more than 1000 testcases which use IO without
>> explicit file names, ~150 use scratches (a temporary file named
>> gfortrantmp + 6 extra char is created) and the others, which only
>> specify the io unit number, use a file named fort.NN with NN the
>> number of the io unit. We see conflicts on these generated files, as
>> lot of testcases use the same number, the most used are:
>>
>> 10 => 150
>> 99 => 70
>> 6  => 31
>> 11 => 27
>> 1  => 23
>>
>> I started to change the testcases to use scratches when it is possible
>> and before finding that there is that many to fix, and I also had
>> conflicts on the generated scratch names.  The options I see to fix
>> that are:
>>
>> 1- Move all these testcases into an IO subdir and change the testsuite
>> to avoid parallelism in that directory.
>> 2- Use scratches when possible and improve libgfortran file name
>> generation, I don't know well fortran but is it possible to change the
>> file name patterns for scratches and io unit files ?
>
> For the "io unit files", I don't think we can change it. fort.NN is a
> common convention, and I suspect a lot of people rely on it. As for
> scratch files, I'm surprised you're seeing conflicts. At least on
> Linux/POSIX targets it should use mk(o)stmp(), which should make sure
> a unique file name is chosen and opened with exclusive access
> (O_CREAT|O_EXCL). Furthermore, on POSIX targets it unlinks the file
> immediately after opening it, further reducing the possibility of any
> conflict. So what target are you compiling for?

OK if people rely on the fort.NN name pattern I understand it can't be changed.

We're validating on x86, ARM and AArch64 linux targets and we
experienced random issues with backslash_1.f90, given the nature of
the testcase and the other issues with fort.NN conflict I assumed it
was the same kind, but it's most likely due to some instabilities we
had in our lab several months ago, as we have no more issues with it
now, and my manual experiment of  mk(o)stmp() usage didn't manage to
create conflicts.  So, Thanks for the clarification and sorry for the
false alarm on scratches.

>> 3- Change the io unit numbers used, as it was suggested on irc, but I
>> find it a bit painful to maintain.
>
> Is it really that much more work than 1)? If you don't want to keep
> track of a count, just generate a random number between 10 and 2^31,
> and use that as the unit number? Should at least make conflicts rare.

I don't think it is more work to fix the actual testsuite, my point of
view is that 1) will guarantee that we will not have any conflicts no
matter which IO units or temporary file names are or will be used,
whereas if we keep them as it, with changing the numbers, maintainers
and contributors will have to take care of choosing an IO unit or a
temporary file name that is not already used in the testsuite.

Thanks,
Yvan