Re: cpplib: locale-sensitive lexing

[Neil Booth <neil at daikokuya dot demon dot co dot uk>]

Zack Weinberg wrote:-

> All the users that are used to being able to get something sensible
> out of raw diagnostic output dumped to a terminal will just *love*
> this idea.

Well, it would be the same they get now for most charsets at least,
but point taken.

> Several different approaches:
> 
> 1. The tactic you were using, of scanning normally until a non-ASCII
>    character is encountered, should work just fine with iconv.  It's a
>    slightly different interface, is all.
> 
> 2. If we require all files which contain non-ASCII codes to be marked
>    in band (which I am in favor of), then converting the entire file on
>    load should be acceptably fast.

Well, non-basic charset rather than ASCII.  ASCII control codes SI and
SO (shift-in and shift-out) are used as shifts in SJIS I believe.

> 3. As a refinement of tactic 2, we could handle general UTF-8 inline;
>    it's easy enough to scan and convert.

OK.  I'm in favour of defaulting to an appropriate charset from the
user's current locale; I think it would be a shame if GCC were the
only major command-line utility that didn't do this.

So, suppose we adopt approach 1. for the moment (and possibly do
something for the others later).  How can we portably get the correct
text string for iconv from the user's locale, or can't we?  (I assume
you would have UTF-8 as the destination charset).  Or if we can't
guarantee that, can we get something that would work for most systems
most of the time?

If not, I'm going to give up on this as it's just too much of a PITA
and let someone else come up with an implementation.

Neil.