Proposed patch for 4 byte unformatted record delimiters

[Rob Ratcliff]

Hi Bud,
 
> the last time i looked at the fancy 'expandable
> approaches', you had to buffer the whole thing in
> memory before writing because you did not know if it
> would be a 32 or 64 bit record marker until you saw
> all the data.
> 
> for a 10K or 100K record, who cares; but for a GB of
> data it is a big deal....not only with respect to the
> time it takes, but also for how much virtual memory it
> consumes.


I'd advocate only using 4 byte or 8 byte markers
in a given file. If we go with the simple compile
time option with no support for continuation (which
I'd be happy with in the short term), then
the user just needs to know (as he would now) not to 
write out records that have lengths beyond what a 
signed 4 byte or 8 byte int can handle. (When I created
the original patch, I just replaced areas where you
looked up the size of gfc_offset to use as the delimiter size
and replaced them with 4 defined by a constant in an include file. 
Of course, the compile time option would just make the 
constant a user-defined value.)

If the continuation model was supported for 4 and 
8 byte (wouldn't want to restrict future users to 
< 2000 quintillion record sizes :-) )
delimitters then it'd be just a matter of setting the sign
as well as the record length to indicate that the 
next record continues.

At least for my use cases, I'd be working with codes that
were compiled to support 4 byte or 8 byte delimiters, but
not both.

Thanks for the interest!

Rob