Comments on modernizing transput (== I/O) in Algol 68

[Nelson H. F. Beebe]

Chris Hermansen in a post yesterday identified many important
points about a modern I/O system for Algol 68.

I commented yesterday that the (page, line, character) tracking of the
"Revised Report on the Algorithmic Language ALGOL 68" (1975) is
poorly matched on modern filesystems, and suggested that perhaps we
ought to abandon van Vliet's transput code implementation in favor of
modern filesystems primitives.

Here are some thoughts:

* Unix introduced the fundamental design idea that ALL files should be
  considered to be byte streams of ZERO or more bytes; any further
  structure, such as lines, pages, text versus binary, embedded data
  structures (C struct, Fortran records, database records) are a
  matter for higher level software, and are completely unknown to the
  I/O system.

  As an example, in Unix, any file can be copied with a simple loop
  using pairs of fgetc() and fputc() calls.

  In DEC VAX VMS, this was impossible, because different I/O
  primitives were needed for each file type.  For example, a text file
  could have lines wrapped by CR and LF, or terminated by CR, or
  terminated by LF, or terminated by CR LF, or represented as a
  two-byte length followed by that many characters (possibly rounded
  up to an even number to fill up a 16-bit word, with a trailing
  padding NUL), or ...

* The Unix stat(), fstat(), and lstat() primitives provide access to
  file metadata (run "man fstat" for details) that includes file type
  (normal, symbol link, device node, ...), user/group/device
  identifiers, size, blocksize, block counts, timestamps (access,
  modification, status change), access flags (read, write, execute,
  sticky, setUID, setGID, ...), ownership (user, group), ...

* While software can open a file by name, it can also receive an open
  file handle (e.g., for /dev/std{in,out,err}) for which it is
  impossible to determine its filename, because the kernel does not
  save that information, or because no named physical file exists
  (pipes, sockets, network connections, Linux /proc files, ...)

* Although some types of files are readable, the stat() family may be
  able to provide only a subset of the normal information, such as for
  Unix input pipes (size unknown), Linux /proc/xxx files (0 size, but
  when read, may have more bytes), WebDAV files (size unknown, because
  the data stream may be generated dynamically by a remote Web server
  as it is read), and files from foreign filesystems that have fewer
  recorded attributes, or more attributes (access control lists,
  character set indicator, ...), than normal Unix files have.

  Older Apple Mac OS filesystem files came in pairs: a data fork with
  user-visible bytes, and a small resource fork with file metadata.
  For access from Unix, software might have to deal with both members
  of the pair.

* The low-level view of files as byte streams matches 8-bit character
  set (ASCII, ISO-8859-n, ...) text files well, but is an issue for
  larger character sets with fixed-length character encodings (UTF-16
  and UTF-32), or variable-length encodings (UTF-8 expected by Algol
  68).  When low-level code reads a byte, it could be part of a
  multibyte encoding, possibly split over a real filesystem block
  boundary, or a pipe block boundary.  Character reconstruction then
  has to be a higher-level job.

* Foreign filesystems may make a distinction between text and binary
  files: a Unix process can open and read both, but might not be able
  to create an output file correctly: the final "mode" argument of
  open() and openat() offers numerous choices (see "man -s 2 open").

* In Unix, pathnames are case-sensitive, and may contain any byte
  except NUL; filenames cannot have either slash (/) or NUL.  In
  pathnames, names may be separated by one or more consecutive
  slashes, considered equivalent to a single slash.  Other filesystems
  (such as on Microsoft Windows and older Mac OS) may be
  case-insensitive, and may or may not be case-preserving.  Lettercase
  treatment in pathnames may in addition be a filesystem mount option,
  such as for the Apple HPFS filesystem, and for ISO 9660 CD-ROM
  filesystems.

* Unix directories can be accessed via the opendir(), readdir(), and
  closedir() calls, allowing software to traverse a filesystem
  tree.

* Unix files can be randomly accessed with the fseek(), ftell(),
  rewind(), fgetpos(), and fsetpos() primitives.  These are needed,
  for example, in PDF file viewers, which use a PDF table near the
  start or end of the file, to locate font and embedded file
  resources, and document page start locations, for rapid navigation.

* Unix filesystem access is based on FILE structures or small integer
  file descriptors. A more general design of some recent languages
  instead uses the idea of data sources and sinks, allowing those to
  be traditional files, or in-memory strings, or network/socket
  connections, or produced/consumed by function calls, or ....

Modern Algol 68 needs to provide similar primitives to those available
to C programmers for all of those points; without them, Algol 68
cannot be used for systems programming, or general filesystem access.
That would cripple its use for major programming projects in Unix.

The GNU Pascal compiler, gpc, (dropped after gcc-7, and no longer
offered by most Unix distributions) had an extensive set of library
extensions to make systems programming possible in Pascal.  The Free
Pascal compiler, fpc, remains available for many systems, and likely
also has many library extensions for I/O and system calls.

An Algol 68 design for Unix I/O support should certainly be aware of
that earlier Pascal work, and learn from it.  The large Java and C#
language libraries have also dealt with these issues, and may provide
useful ideas for ga68 work. 

The simplest approach, however, is likely to be using similar
interfaces to all of the C I/O library functions, because (a) they are
familiar to C/C++ programmers, (b) they have been developed and proven
themselves over a half-century of use, and (c) they are
internationally standardized in POSIX.

-------------------------------------------------------------------------------
- Nelson H. F. Beebe                    Tel: +1 801 581 5254                  -
- University of Utah                                                          -
- Department of Mathematics, 110 LCB    Internet e-mail: beebe@math.utah.edu  -
- 155 S 1400 E RM 233                       beebe@acm.org  beebe@computer.org -
- Salt Lake City, UT 84112-0090, USA    URL: https://www.math.utah.edu/~beebe -
-------------------------------------------------------------------------------