Re: thoughts on martin's proposed patch for GCC and UTF-8

[Paul Eggert <eggert at twinsun dot com>]

   Date: Wed, 9 Dec 1998 23:27:17 +0100
   From: Martin von Loewis <martin@mira.isdn.cs.tu-berlin.de>

   > * GCC should represent non-ASCII identifiers using the locale's
   >   preferred multibyte encoding

   I assume you talk about error messages, here?

I'm talking about every place that GCC outputs identifiers.  This
includes error messages, assembler output, and other auxiliary text
output (e.g. -dM output).

   >   If the locale cannot represent a particular Unicode character,
   >   GCC should store it in a canonicalized escape form (e.g. the
   >   locale's encoding for \u with lowercase alpha digits if it fits
   >   in 16 bits, \U with lowercase alpha digits otherwise); this is
   >   along the lines of what draft C9x suggests.

   Can you elaborate? What is the locale's encoding for \u, if not "\u"
   literally?

If it were up to me, it would always be \u or \U as specified above.
We have to be a bit careful, though, as we can't just splice
e.g. "\u1234" into a multibyte sequence; we may need to surround the
"\u1234" with bytes that bring us to the initial shift state and back
again.

The platform may have its own idea of the canonicalized escape form.
ASM_OUTPUT_LABELREF could arrange to convert to the platform's format.

   what exactly is a UTF-8 locale?

a locale that uses UTF-8 encoding for multibyte characters, and
(obviously) that uses the Unicode character set.

   E.g. would the "C" locale qualify?

No, typically "C" uses plain ASCII with no multibyte chars, though it
can and sometimes does use something else (e.g. ASCII subsets).

   >   One way to translate from Unicode to non-UTF-8 is to have GCC use
   >   the iconv function if available.

   There might be a problem with the native iconv: What if it doesn't
   have conversion to Unicode, or what if we don't know what the name of
   Unicode is in a particular iconv implementation (we always know for
   glibc iconv, i.e. gconv).

Yes, but that's just one thing to configure.  And the default value,
"UTF-8", will probably work on most hosts.

There's also the problem of getting the name of the encoding for the
current locale.  The XPG4 way of doing this is `nl_langinfo (CODESET)'.
This is supported by glibc 2.1 and by recent Solaris versions.
Systems that don't support this will have to be supported ad hoc
(if at all).

   I'd like to ask whether incorporation of gconv into egcs would be
   acceptable to the egcs maintainers; if this is the case, I will
   work towards such an incorporation. Having gconv available would
   simplify cross-compilation.

Since egcs already maintains its own idea of S-JIS etc. I suspect that
there'd be no objection to also maintaining its own idea of what the
translation tables should be, for hosts that don't already have it.

   > * Given the above, I don't see the need for TREE_UNIVERSAL_CHAR.  The
   >   identifier should be stored using the locale's multibyte chars as
   >   suggested above (with canonical escapes if needed), and output
   >   as-is, just as identifiers are now.

   Well, it is needed for name mangling. Name mangling (in object files)
   needs to be independent from the user's locale; otherwise you can't
   link libraries produced by somebody else.

I see the need for mangling, but I don't see why TREE_UNIVERSAL_CHAR
is needed.  When outputting a name, you don't need to have a separate
flag specifying whether whether the identifier contains \u; you can
just inspect the identifier string directly.  This would be
ASM_OUTPUT_LABELREF's job.

Also, I assume that once the patch is generalized to non-UTF-8
locales, it won't be just the \u and \U escapes that require mangling.
If the goal is to link libraries that were built in other locales,
then we'll also need to mangle the non-UTF-8 multibyte chars.  Again,
this doesn't sound like a high-level concept that needs to be in the
parse tree -- it's just a low-level thing that can be done on output.

Perhaps it's just an efficiency thing?  If so, then this should be
made a bit clearer, and the flag name changed to
TREE_NAME_NEEDS_ASCIIFYING or something like that, with the other
identifiers named changed accordingly.

   jc1 already defines object files to use UTF-8. It seems that jc1
   integration is a objective for cc1plus, so we need to keep that fixed.

If the compilation locale uses, say, Shift-JIS, then the assembly
language text file should use shift-JIS, as this is what the
programmer's tools will expect.

If the object-code standard is to use UTF-8 names, then I suppose the
assembler can convert to UTF-8.  (Object code isn't text, so the usual
rules about text locales don't apply to it.)  But this would mean that
the assembler would have to understand character conversion, which is
a unwanted complication.

However, if jc1 is meant only to be used in UTF-8 locales (which seems
likely), then we needn't worry about this.  We just tell people that
they have to use an UTF-8 locale if they want to use jc1 with non-"C"
names, because jc1 object files must use UTF-8.  This would be an
understandable restriction, and it means we could avoid having to do
the translations in either the compiler or the assembler.

   >   People who use multibyte chars in identifiers will expect them to go
   >   through to the assembler; if the assembler doesn't support them,
   >   they'll understand the assembler's error message.

   This is certainly true for C, it does not hold for C++. In C++, you
   can produce output even if the assembler does not support non-ASCII in
   labels.

Sorry, I don't understand this point.  If you're saying that C++
mangles non-ASCII identifiers into ASCII labels, but C doesn't, then I
don't see why that should be: there's no reason in principle that C
couldn't or shouldn't use the same sort of mangling.

If the assembler requires some form of mangling from non-ASCII
identifiers into ASCII labels, then shouldn't this be
ASM_OUTPUT_LABELREF's job, or something like that?  I don't see why
the issue is specific to C++; it sounds like it's general to all
languages with non-ASCII identifiers.

   >   Also, the configure.in test for HAVE_GAS_UTF8 has UTF-8 in it.  This
   >   won't work with older shells that don't allow UTF-8.  It's simpler if
   >   we just remove HAVE_GAS_UTF8.

   What systems don't support 8-bit characters as arguments to echo?

I've run into shells that use the top bit for their own purposes.

And, even if such shells are discounted, it's a bit odd to use UTF-8
in configure.in without labeling the file.  My Emacs (20.3)
misidentified the file as being ISO Latin 1.  It'd be better if you
rewrote the configure.in test in ASCII, so we didn't have to worry
aboug gotchas like this.  It should be fairly easy to do this with tr.

   C++ does not distinguish between non-ASCII digits and letters.

Really?  Suppose I write the preprocessor line

#if X == 1

where X is some Japanese identifier, but I make the understandable
mistake of using a FULLWIDTH DIGIT ONE (code FF11) instead of an ASCII 1.
What you're saying is that the preprocessor is obliged to treat this
line as if it were

#if X == 0

because undeclared preprocessor identifiers default to zero?

This seems to me to be asking for trouble; it's a common mistake in
Japanese text.  If C++ requires this, then I suggest that C++ by
default should warn about identifiers beginning with digits.  (It also
means yet another difference between the C and C++ preprocessors,
sigh.)