Re: -frecord-marker

[Janne Blomqvist <Janne dot Blomqvist at tkk dot fi>]

On Wed, Jul 26, 2006 at 02:04:16PM +0100, Adrian Umpleby wrote:
> > On Fri, Jul 21, 2006 at 04:45:07AM -0700, Bud Davis wrote:
> > > on some file systems those seeks will be slow. very
> > > slow.  and of course they will be worse the bigger the
> > >  records :)
> > 
> > In an absolute sense you're correct since longer seeks take a longer
> > time,
> 
> Actually, having looked at the code, I don't think it makes any practical difference at all, since
> there are usually no extra seeks to perform over what is already done, and when there are, it's
> negligible compared to the writing.

Indeed; I wasn't talking about the Intel scheme vs. some other record
marker scheme, but more generally.

> What happens at the moment is that a record marker 'gap' is placed at the start of a record (since
> we have no idea beforehand how long the record will be). Once the record is finished, the ending
> record marker is written, with the appropriate record length, and it has to seek back to that
> 'gap' to fill in the length there too. Then it seeks forward past the final marker again (ready to
> write the next start of record marker). So, that's two seeks to perform already, no matter how
> long the record is.

Yes, just like I explained in a previous message to you a week ago:
http://gcc.gnu.org/ml/fortran/2006-07/msg00283.html

> For the Intel record marker format, there is no difference at all for records <2GBytes (which
> includes your direct_io_5.f90 case?) -you can describe it as 'worst case', but it's still no
> different to what gfortran has to do right now.

Again, I meant worst case as in when seeks hurt overall throughput
most, not anything specific to Intel record markers. Sorry for the
confusion.

> So, the extra work done in this case (only when record is >2GB) is to perform an extra pair of
> seeks after every 2GB. The 'worst case', then, occurs with records which are just a little over
> 2GB - it then has to do four seeks rather than two. Compared to the time to write that 2GB, I
> doubt there will be any noticable difference at all...

Yes, just like I said in my previous message; specifically:

"""

With the Intel scheme we'd be limiting ourselves to 2 GB subrecords,
meaning that every 2 GB we would need to seek back and fix the
marker. That's not a problem, unless we're talking about tape io.

"""

> Assuming my understanding of the gfortran code is correct, it seems to me this is the main reason
> why the Intel method really is the best of the bunch to implement.

I agree. 

> (And I'm struggling to see any
> decent way to implement the HP method - how on earth can you go back and insert a few more bytes
> before the record to form a new 64 bit record marker after already writing that >2GB of record?
> -How can you avoid shifting it all along?!

You can't. Unless you happen to know the size of the record before you
start writing, which incidentally is how the HP compiler manages to do
it (see http://gcc.gnu.org/ml/fortran/2005-02/msg00270.html ).

And even if we were able to figure out the record size beforehand, I
still think the Intel scheme would be better from a "not painting
ourselves into a corner" perspective. Absolutely requiring us to know
the size beforehand is a pretty big constraint, although it would be a
usable optimization applicable to any record marker scheme if we knew
it, at least in the majority of cases (fall back to the current method
of seeking back for corner cases).

> Is it possible to 'trick' it in some way by having a
> zero length record [i.e. just two 32-bit markers, containing zeros] before each true record, which
> you can then use to have the required space for a 32 bit non-marker [with MSbit set] and a 64 bit
> record marker if required afterwards??)

That would be no more compatible with current 32-bit record marker
formats than the 64-bit marker gfortran uses today.

-- 
Janne Blomqvist

Attachment: pgp00000.pgp
Description: PGP signature