Re: thoughts on martin's proposed patch for GCC and UTF-8

[Paul Eggert <eggert at twinsun dot com>]

   Date: Sun, 20 Dec 1998 21:29:41 -0700 (MST)
   From: Richard Stallman <rms@gnu.org>

       Perhaps it would be better for GCC to autodetect the character set and
       encoding, much as Emacs already does.

   Autodetection is limited; in Emacs it requires (in some cases)
   preference information from the user.  For example, Emacs cannot
   distinguish between Latin-1, Latin-2, Latin-3, Latin-4 and Latin-5.
   There is no way to distinguish them automatically, because they use
   the same set of valid bytes.

There are two related issues here:

1. Can autodetection work well enough to support non-"C" characters in
   strings and identifiers?

2. Can autodetection work well enough to support \u escapes as well?

Your example suggests that the answer to (2) is ``no''.  But this
already follows from the fact that a C program could be written
entirely in the "C" character set with \u escapes, and autodetection
can't possibly determine the multibyte encoding of such a program.

I was thinking more about case (1), which I think will be more common
in practice.  I suspect that autodetection could work reasonably well
for \u-free programs.  For example, GCC needn't worry about the
distinction between Latin-1 and Latin-2 if there are no \u escapes,
since it needn't worry about whether the byte 0xb5 corresponds to
MICRO SIGN or to some other character.

       all the pragmas required by draft C9x affect the program's meaning

   The committee is making a foolish decision.

The committee is also requiring _Pragma("FOO") to have the same
meaning as #pragma FOO.  _Pragma("FOO") can be output by macros.
Does this overcome your objection to pragmas?

       * If the program text is converted from one encoding to another, its
	 #locale will become incorrect.

   If GCC is going to depend on the locale, you will have to specify the
   locale for your files.  Regardless of how you specify it, with #locale
   or with --locale or with an envvar, in any case there is a risk you
   might forget to change the specification along with the file.

I have a shorter and a longer answer to this.

The shorter answer:

	GCC currently doesn't have directives like this:

	#character-set ASCII
	#character-set EBCDIC

	because they're not needed; people who compile in EBCDIC environments
	already know about these issues, set things up appropriately, and
	would find those directives to be a pain to maintain.  GCC (and other
	compilers) have survived all these years without character-set
	directives, even though they solve roughly the same problems that
	#locale directives would solve.  This suggests that GCC doesn't
	need #locale directives either.

The longer answer:

	There is a risk to mis-specifying the locale, yes, but in practice my
	experience is that the risk is smaller if the locale is part of the
	environment.

	In our company, when we import files from other sources, we typically
	transliterate them to an encoding suitable for our preferred working
	locale.  This is the only plausible way to do things; otherwise, few
	of our text-processing tools would work.  Even Emacs supports only
	_some_ of the Japanese encodings that we import -- e.g. it doesn't
	support UTF-8 or DBCS.  Most other tools support only one character
	set and encoding at a time, and it is set from the locale environment
	variables in the usual way.

	If #locale were part of the source, we'd have more work to do, since
	we'd also have to munge the #locale directives of imported sources.
	This would be doable, but it would be a hassle, particularly when
	trading patches with our correspondents who use different encodings.
	I can easily see where people would screw this up.

	In contrast, if the locale is part of the build environment, we
	needn't worry about munging anything.  We must set up our build
	environment correctly, but that's OK -- we also must set up our
	PATH correctly, etc., and setting up the locale correctly is something
	that everyone versed in software internationalization and localization
	already knows how to do.


   use of #locale avoids the danger that you will simply forget
   to specify the right locale,

No, it doesn't avoid the danger.  You can specify the wrong locale
just as easily, if not more easily, with #locale -- e.g. see the
transliteration scenario in my longer answer above.

       * Locale names aren't very portable.  E.g. Solaris uses "ja" for
	 EUC-JIS whereas Unixware uses "ja_JP.EUC".

   This is a real issue.  I see three possible solutions.

   1. Define our own system-independent names for (some) locales.

I'd rather not do this -- it will be a maintenance hassle.  But if we
must do it, we should steal code from glibc rather than reinvent the
wheel (as is done in the current GCC2 and EGCS snapshots).

   2. Allow specification of several locale names, and GCC will use
   the first one that is meaningful on the system in use.
   3. Allow specification of several locale names, each associated
   with a host system type.

These are also maintenance hassles, for several reasons.  E.g. "ja"
means different things on different hosts.  I don't know which locale
names map to which encodings on which hosts, and keeping track of this
info will be tedious and quite error prone.