Re: -frecord-marker

[Janne Blomqvist <Janne dot Blomqvist at tkk dot fi>]

On Fri, Jul 21, 2006 at 11:11:06AM +0100, Adrian Umpleby wrote:
> At the moment I know of at least four (implemented) schemes for the record length marker:
> 
> 1) 32 bit only - the way it was always done, 'cos we're never gonna need >=2^31 bytes :)
> 
> 2) Intel - 32 bit, but if MSB is set, it means the next 'chunk' after this one is a continuation
> of the same record: this allows a potentially unlimited record length, and is backward-compatible;
> 
> 3) HP - 32 bit to start with, but if the 32 bit marker is set to hex value FFFFFFFF (or just has
> MSB set?) then a 64 bit marker follows: this is also backward-compatible, and could potentially be
> extended in a way that remains backward-compatible;
> 
> 4) 64 bit only - the way gfortran does it (on most modern machines), *not* backward-compatible,
> and not compatible with *any* of the other schemes, old or new. (And we're never gonna need >=2^63
> bytes...)

Yes, those are the ones which have been discussed. As a minor nitpick,
xlf also supports scheme (4), although it defaults to (1).

> Doesn't this scream out that the problem is precisely one of standards - more specifically, lack
> thereof! (Someone correct me if I've missed it somewhere...)

It screams out that someone has had the misconception that Fortran
disk layouts have been standardized, and has thus mistankenly come to
rely on it. As has been mentioned before, the solution to portable
binary io is a library such as netcdf.

> If you want me to, I'll be happy to add my own idea for yet another scheme, which is both
> backward-compatible and extensible  :)

Feel free. Currently I think the Intel scheme is the best, but
suggestions for improvements are always welcome.

> The major problem with the schemes other than 1 & 4 above (and this would also apply to my own
> scheme, and probably any that wants to remain backward-compatible) is that they require knowledge
> of the length of a record before writing its record marker. AFAICT, this information is not
> available in the current gfortran code at the time the record marker is written, and it's not at
> all obvious to me what the best way is to find it. If I've got this right, it means implementation
> of Intel & HP record marker schemes will not be straightforward at all (maybe someone who knows
> the code better can help verify).

AFAICS it's not necessary to know the size of the record before
writing it with the Intel scheme. When running out of space in one
subrecord, just seek back and fix the first record marker, seek
forwards and start the next one. Just like gfortran does it today. It
does cause some extra seeking, but changing that is unfortunately
rather complicated.

> Another way of dealing with this whole problem might be to have a conversion tool. However, with
> the RECORD_MARKER tag, it becomes possible to actually write that tool using gfortran - and then
> everyone else can forget the tag ever existed  :)

With the F2003 ACCESS='STREAM' it's also possible to write the tool
using Fortran; Jerry Delisle is AFAIK working on implementing it. Or
if you needed the conversion tool yesterday, it should be simple to
write in C for that matter.

> I hope that helps to explain the issue we're faced with here!

Of couse, no matter what is done or not done, somebody will be unhappy.

-- 
Janne Blomqvist

Attachment: pgp00000.pgp
Description: PGP signature