Re: thoughts on martin's proposed patch for GCC and UTF-8

[Paul Eggert <eggert at twinsun dot com>]

   Date: Mon, 21 Dec 1998 23:28:09 -0500
   From: Zack Weinberg <zack@rabi.columbia.edu>

   >(2) GCC uses the iconv function to translate from the input multibyte
   >    encoding to UTF-8 internally (for identifiers), and to determine
   >    character boundaries (in strings and comments).

   To do this we'd need to translate the entire file to UTF-8 in order to
   know where identifiers begin and end, and then translate strings
   back.

We can avoid this problem by using iconv in ``byte-at-a-time'' mode.
I.e. we can use iconv to discover the minimal nonempty sequence of
input bytes S such that S is a multibyte character string, and such
that the first char after S is a "C" char.  If we find such an S in a
string, we copy its value through unchanged; if we find it in an
identifier, we translate it to UTF-8.  This would mean we wouldn't
have to translate the entire file to UTF-8.

As an optimization, we don't need to call iconv at all in the common
case where the input file uses only the "C" subset of ASCII.  This is
because such files cannot contain multibyte chars.  We need to call
iconv only if the input contains non-"C" bytes (e.g. bytes with the
top bit on, or the ESC character).

It might be nice if there was something faster than invoking iconv in
byte-at-a-time mode, for files that contain lots of multibyte chars.
E.g. it might be nice if we could use an efficient primitive that acts
like iconv, except it stops translating when it finds a "C" char.  If
necessary to improve performance, we can add such a primitive to
glibc, and use it if it's available.