Re: thoughts on martin's proposed patch for GCC and UTF-8

[Paul Eggert <eggert at twinsun dot com>]

   Date: Mon, 21 Dec 1998 18:45:09 -0800
   From: Per Bothner <bothner@cygnus.com>

   we want to support linking together source files written in
   different locales...  The only plauible contender for such a common
   character set is UTF-8.

OK, how about this proposal?  I've tried to formulate it to address
everybody's concerns:

(1) The input character set is determined by #pragma charset FOO
    (or _Pragma ("charset FOO")) directive, compile-time option, or
    environment variable (in that order).  For the last alternative,
    the default is to use setlocale (LC_CTYPE, ""); nl_langinfo
    (CODESET) if these two functions are available.  The default
    is UTF-8.

(2) GCC uses the iconv function to translate from the input multibyte
    encoding to UTF-8 internally (for identifiers), and to determine
    character boundaries (in strings and comments).  If the
    implementation doesn't have iconv, GCC normally supports only
    UTF-8; however, if the installer wants to build a compiler that
    knows about other encodings (e.g. for cross-compilation), we
    supply an easy way to use glibc's iconv.  We can then remove the
    existing local_mblen function and friends, as they're no longer
    needed.

(3) GCC transliterates each \u escape in a string to the string's charset,
    which is specified as described in (1) above.

(4) After the translation in (3) (and after processing the other
    escapes like \n), GCC copies the contents of strings straight
    through to the assembler, if possible.  As is currently the case,
    characters like \ and " that need escaping are escaped.  However,
    a new feature is that if a string contains troublesome multibyte
    characters (e.g. the characters contain the bytes for ASCII \ or
    "), then those characters are output using octal escapes for each
    byte.  Similarly, if there is a string of multibyte characters not
    in the initial shift state that contains a \ or " byte, the entire
    string is output using octal escapes.

(5) GCC transliterates all identifiers to UTF-8 for the assembly
    language output.  If the input character set is a superset of UTF-8
    (e.g. ISO-2022-JP), then the extra information is lost.  If the
    assembler doesn't support UTF-8 identifiers, GCC transliterates
    identifiers to some ASCII escape sequence representing the UTF-8
    identifiers.

(6) GCC transliterates all identifiers to the working charset for all
    other output (e.g. diagnostics).

Here are some properties of this proposal:

* If the input file uses UTF-8, then the assembly language output
  uses UTF-8 as well.

* If the input file is a text file that does not use \u escapes, and
  does not use multibyte characters in identifiers, then the assembly
  language output is a text file that uses the same encoding.  This
  should accommodate existing practice reasonably well.

* The assembler needn't know about encodings.

* You can link together source files written in different locales,
  since all the identifiers are transliterated to some encoding
  of Unicode.