RFC Wide Characters in I/O

[Jerry DeLisle]

Here are my thoughts:

Modify the front-end in trans-io.c to build a call to a new function called 
transfer_wide_character. This function is used if we have an ENCODING= specifier 
or a wide character to transfer. This will retain the existing 
transfer_character call to maintain compatibility.

Repeating myself, the new function will be used for kind > 1, while 
transfer_character is retained for kind=1 and hollerith as is.

The downstream functions in the library such as list_formatted_read, 
list_formatted_write, formatted_transfer, unformatted_read, unformatted_write, 
already accept a kind parameter which transfer_character sets to 1.

The further downstream functions for character I/O we will then modify to use 
the kind parameter that is passed in.

If we are given a wide character to transfer as unformatted, we simply transfer 
all the bytes as is. (I will confirm with the standard on this.)

If the user has specified an ENCODING="default" and the kind is 1, we do what we 
do now and transfer as 8bit (mostly ASCII).  If kind is greater than 1, I 
suggest we transfer each byte as is.  So for kind=4, we would transfer 4 bytes. 
This would enable doing some packed 4x1 byte character stuff.  I think the 
standard would allow this and thats why "default" is so loosely defined.

(Note: It is my understanding, in reading about UTF-8, that it is internally 
represented by a fixed width 4 byte hexidecimal number which when transfered to 
an external file, is translated on the fly into a variable width encoding.  When 
reading back in, it is translated back to the 4 byte code.)

If the user has specified an ENCODING="UTF-8" and the kind is less than 4, 
strictly speaking, thats an error, but we can give a warning and assume a 
truncation. In other words, the upper most bytes of a 4 byte representation are 
assumed to be zero and we transfer after translating hex-to-utf8 (or likewise 
utf8-to-hex).  If on reading  we encounter an encoding that will not fit in the 
given character kind, we throw a runtime error "Character encoding exceeds 
KIND=2 at %L" or similar.

For kind=4 and ENCODING="UTF-8", we do a complete conforming translation of the 
4 byte hex to/from the variable width UTF-8 encoding.

Comments welcome,

I will start on this if we are in agreement.  Timing wise, I think this should 
go in after Jannes part 2 of getting rid of the stream_alloc stuff.

Regards,

Jerry