Re: cpplib: locale-sensitive lexing

[Zack Weinberg <zack at codesourcery dot com>]

On Wed, Oct 24, 2001 at 10:12:20PM +0100, Neil Booth wrote:
> This patch implements locale-sensitive lexing as (by my reading) the
> C99 standard requires it.

I would prefer an approach which made no use of Cx9 multibyte
character functions.  Specifically, the concept of "locale" is far too
slippery.  We run the risk of having code compile correctly on one
system and not on another because the library implementers have
decided that "ru_RU" means something different.

The Java front end appears to ignore locale and use iconv and
character encoding names.  This seems much more appropriate to me.

I also think it's short-sighted to implement a solution which does not
allow for the source character set to vary per input file, or for the
execution character set to be different from the source character
set(s).  We know we will need these things.  We don't have to
implement them now, but let's not implement something that will have
to be thrown away and redone later.

A related problem we need to consider is the width of non-ASCII
characters.  For instance, most Chinese ideograms take up two columns
each.  In order to get column positions right, we have to check what
the width of each character is.  C99 has wcwidth(), but again I would
prefer we avoid anything that depends on locale.

zw