This is the mail archive of the
fortran@gcc.gnu.org
mailing list for the GNU Fortran project.
Re: [Patch, Fortran ] PR 50016: Slow Fortran I/O on Windows and flushing/_commit
On Mon, Oct 17, 2011 at 19:03, Tobias Burnus <burnus@net-b.de> wrote:
> Hi Janne,
>
> On 10/17/2011 05:30 PM, Janne Blomqvist wrote:
>>
>> On Mon, Oct 17, 2011 at 15:49, Tobias Burnus<burnus@net-b.de> Âwrote:
>>>
>>> This patch adds a call to _commit() on _WIN32 for the FLUSH subroutine
>>> and
>>> the FLUSH statement. It removes the _commit from gfortran's buf_flush.
>>
>> Like I argued in this message
>> http://gcc.gnu.org/ml/fortran/2011-10/msg00094.html, I think this is a gross
>> mistake.
>
> [...]
>
> And I think it is a mistake to not make the data available to other
> processes as it is indicated by the Fortran 2008 standard:
>
> "Execution of a FLUSH statement causes data written to an external le to be
> available to other processes, or causes data placed in an external file by
> means other than Fortran to be available to a READ statement. These actions
> are processor dependent."
>
> Thus, I think it makes sense for FLUSH to call _commit on Windows.
I'm not actually sure we can draw such conclusions. What we know is
that metadata updates to the directory are delayed. It wouldn't
surprise me if opening a file (which presumably is a atomic operation
in order to avoid race conditions just like on POSIX) forces the
kernel to sync metadata of other handles to the same file.
> If you don't want to have a slow down: Simply do not call FLUSH.
There are legitimate reasons flush which do not entail storing dirty
data to stable storage. Which is why things like fflush() and fsync()
are separate.
>> libgfortran should not require _commit nor fsync in any situation. Those
>> calls are useful for writing databases and other applications which must
>> make data integrity guarantees, and are prepared to pay the performance cost
>> associated with it. It's absolutely not something a language support library
>> should do unless the language spec explicitly requires such data integrity
>> guarantees.
>
> Well, Fortran does not need to write the data to the file, however, the
> purpose of FLUSH is that I can, e.g., run execute_command_line with the file
> the program just has written. It will work on Unix/Linux but not on
> MinGW/MinGW-w64 without a _commit (or without closing the file).
Do we have any evidence of this failing in reality?
>> That write() would be buffered on windows makes no sense to me
>
> Why shouldn't it be buffed?
Because language runtimes already provide buffered IO so there's no
reason for what is ostensibly a syscall to provide an extra layer of
user space buffering. Secondly, AFAIK once a write is finished, the
data is there even if the application crashes, which suggests that the
data is transferred to the kernel page cache just like on other OS'es.
To clarify, I was talking about a per-process user space buffer, like
C stdio or indeed the internal buffering in libgfortran. Of course,
like any remotely modern OS Windows does aggressive buffering of files
in the kernel using whatever memory is available.
> Typical Windows programs open files with an
> exclusive lock and as Windows never had the pipes and many small programs as
> Unix did, having a per-file-descriptor buffer is easier to implement, avoids
> multi-thread issues and is potentially faster.
The fact that Windows, like many other OS'es, supports things like
shared libraries and shared file mappings, strongly suggests that it
has a unified page cache for file data, just like Linux. Also, an OS
that would waste memory on caching the same file multiple times would
not be performance competitive in today's world.
>> And, while I'm at it, this kind of "relaxed consistency" is not
>> unheard of in the unix world either. Consider NFS, where data and
>> metadata may not be flushed to the server until fsync() or close() is
>> called, or the attribute cache timeout forces the writeout(?), and
>> thus it's possible for clients to have an inconsistent view of a file.
>
> Well, most of the time it works well on the same system:
Yes, most of the time it works just fine. If we're going to do
something with massive performance implications, IMHO we'd better make
sure it works always and not just most of the time.
>> In both cases the remedy is the same; if this kind of consistency matters,
>> the user should close the file or fsync()/_commit() before expecting that
>> the OS metadata is consistent. I think that's a better option than
>> sprinkling _commit() all over the library.
>
> No, for the required consistency, FLUSH is enough (including calling _commit
> on MinGW/MinGW-w64). It makes sure that if the program crashes, the data is
> still there, it makes the data available for other processes.
Like I argued above, I'd be VERY surprised if written data is actually
lost if the application crashes without first doing a _commit. Also, I
find it likely that opening a file will actually provide the correct
up-to-date metadata even though the directory might contain slightly
stale metadata.
> Only if one wants to have complete integrity, one can call fsync. However,
> with NFS, Lustre et al., I am not 100% sure that the data is immediately
> available on all other clients after fsync returned.
With NFS, no. fsync() just guarantees that data is committed on the
server; other clients will notice it only when their attribute caches
timeout and they refetch the attributes for the file.
For Lustre, yes, since Lustre implements a full cache coherency protocol.
> Regarding your patch: I think it does not solve the FLUSH issue.
Why not?
> For the
> file size itself, I think the patch is okay, but frankly, I think for the
> performance it does not really matter which approach is taken.
I do think performance matters. The performance of fsync()/_commit()
is limited by the rotation speed of the HD; with a normal 7.2k SATA
disk one can do maybe 100 fsync()'s per second. A terribly small
number, that is, if one compares to the performance of the CPU and
memory on a modern PC. For comparison, I recall I was able to do
something like 10**6 or was it even 10**7 simple syscalls per second.
> And I do not
> like the test-suite part of your patch.
?? Well, with the rest of the patch the FLUSH is unnecessary, but I'm
sure it won't do any great harm if it's kept.
--
Janne Blomqvist