Re: [Patch, fortran] PR 56981 Improve unbuffered unformatted performance

[Janne Blomqvist <blomqvist dot janne at gmail dot com>]

On Mon, Apr 29, 2013 at 1:46 AM, Jerry DeLisle <jvdelisle@charter.net> wrote:
> OK Janne and thanks for the patch.

Thanks for the review, committed (as well as the system_clock patch).

> What are your thoughts about special casing nul devices/

Hmm, I'm not that eager. It starts to smell of "benchmarketing"..

One thing I have been thinking of which could help would be to
implement a "start of the current record" marker in the buffering
implementation, and when flushing then only flush up to that marker.
Currently when writing small sequential unformatted records what often
happens when looking at strace output is something like

write(3, "\4\0\0\0`\0263I\4\0\0\0\4\0\0\0p\0263I\4\0\0\0\4\0\0\0\200\0263I"...,
8192) = 8192
write(3, "\4\0\0\0", 4)                 = 4
lseek(3, 8810688, SEEK_SET)             = 8810688
write(3, "\4\0\0\0", 4)                 = 4
lseek(3, 8810700, SEEK_SET)             = 8810700
write(3, "\4\0\0\0\20A3I\4\0\0\0\4\0\0\0
A3I\4\0\0\0\4\0\0\0000A3I"..., 8192) = 8192

i.e. the buffer fills up in the middle of a record, we flush it (the
8192 byte writes), but then we have to seek back and forth to fix the
record markers. This also means that we cannot fully buffer
non-seekable files (such as /dev/null) because the records around the
buffer boundaries are corrupted (which of course doesn't matter for
the particular case of /dev/null, but otherwise..). So if this issue
is fixed we could buffer those as well and get essentially the same
performance as for regular files.

E.g. something like

/* Reserve space in the buffer (flush existing data if necessary), up
to some reasonable max size (e.g. 4 KB) unless the size is small and
known upfront (e.g. direct access), and set the current_record_start
marker at the current position. Should be called when preparing a new
record (st_write()). */
void sreserve(int size);

Write the record data as usual...

/* Finish the record, i.e. set current_record_start marker to -1 to
mark that there is no current record. size should be <= reserved size.
Should be called when finishing a write (st_write_done()). */
void scommit(int size);

Well, that's a rough idea. I don't know when or if I'll have time and
motivation to implement it, though..

--
Janne Blomqvist