GCC 4.7.0 Status Report (2011-10-11)

[Tobias Burnus]

Am 14.10.2011 19:37, schrieb Janne Blomqvist:
> On Fri, Oct 14, 2011 at 13:15, Tobias Burnus<burnus@net-b.de>  wrote:
>> On 10/14/2011 10:53 AM, Janne Blomqvist wrote:
>>> I think one could add 50016 as a regression as well.
>>> http://gcc.gnu.org/bugzilla/show_bug.cgi?id=50016
>>>
>>> It seems IO performance in some cases on Windows has tanked since the
>>> 4.6 release (at least by 2 orders of magnitude according to the bug
>>> reporter). I don't have a windows system to test on, so I can't
>>> confirm whether my suspicion that I mentioned in the PR is correct.
>>> But it seems reasonable.
>>>
>>> I think in general, we ought to think what inquiring for the size of
>>> an open file means (the standard doesn't really help here, AFAICS):
>>>
>>> - Is it the size of the file as reported by the OS?
>>>
>>> - Is it the size of the file as seen by libgfortran? (which can be
>>> different than the previous option, due to buffering)
>>>
>>> - Or should we flush the buffer in order to make the OS and
>>> libgfortran sizes consistent at the moment of the inquire execution?
>>> This is what we do now, but for some strange reason it seems to
>>> require the equivalent of fsync() on windows (WTF?), so performance is
>>> bad.
> Sorry, it seems I misspoke. Looking at the implementation
> (libgfortran/io/inquire.c) it seems that what we actually do is
> something like the first of the three options above (i.e. we always
> figure out the size of the file, regardless if it's opened or not, by
> calling stat("path/to/file",...)).
>
> In the testcase inquire_size.f90 we then execute the FLUSH statement
> in order to make sure that the libgfortran and OS views of the file
> size matches. FLUSH then, in turn, causes buf_flush() to be called
> which is the function that calls _commit() on win32.
>
>> How about the following:
>>
>> By default, e.g. for raw_flush and friends, do not call Windows' _commit.
>> That's equivalent to the current Linux/Unix behaviour and undoes the patch
>> for PR 44698.
>>
>> If the user calls INQUIRE on an open file or calls the FLUSH() statement or
>> the FLUSH() subroutine, an explicit _commit is done on Windows. That way,
>> the system buffer is flushed and an access with a different file descriptor
>> to the same file sees the current version. That's the same as one already
>> has on Unix/Linux/Darwin, except that there it happens automatically and
>> that FLUSH does not cause data written to the filesystem (only fsync()
>> does), but that's OK.
>>
>> I think that should solve the problem of PR 44698 and PR 50016 and give a
>> consistent view of the file system without imposing too much performance
>> problems.
>>
>> Janne, what do you think? What do others think?
> I'd prefer to avoid _commit() at all costs, if at all possible; as we
> have seen, it's really slow (since disks are very very slow compared
> to RAM). I just find it hard to believe that there is no way to flush
> user-space buffers to the kernel on windows so that the file data and
> metadata are updated without having to also force the kernel to flush
> dirty data to disk.
>
> If nothing else, maybe we should change inquire for size in
> inquire_via_unit() to use file_length() instead, which would use the
> libgfortran info about the file size instead of the stat() syscall?

Well, I do not worry about INQUIRE - I really do not think that programs 
use it in a loop. Thus, I am happy with either _commit, or an internal 
file-size read, or the current stat without _commit.

In any case, we should remove the _commit from internal/implict buffer 
flushes. Still, for explicit FLUSH statements (or subroutine calls), I 
think gfortran should make sure that the data ends up such that other 
programs can read the file with the current value.

Users should know that FLUSH is relatively expensive - thus, FLUSH 
should only be sparsely used. Would be my patch without the inquire.c 
change be an option for you?

Regarding Windows: I think in the history of Windows, most programs 
opened files with exclusive locks. Additionally, contrary to Unix, under 
Windows one does not have log files which are continuously written to 
and read from other programs. Thus, there was no real need of allowing 
different processes/file descriptors access the file-descriptor's system 
buffer.

Tobias