cpplib: locale-sensitive lexing

[Zack Weinberg]

On Thu, Oct 25, 2001 at 06:32:51PM +0100, Neil Booth wrote:
> Hi Zack,
> 
> One more question about an iconv-based approach.  What would you do
> about preprocessed output, and also column tracking?

Preprocessed output always comes out entirely in UTF-8, and the
compiler has intrinsic support for UTF-8 even if the system doesn't
have iconv or even C89 <wchar.h>.  This means there is never any
problem with shipping a preprocessed file somewhere and interpreting
it there.

Column tracking - we implement our own wcwidth() equivalent that we
know works and which always takes UCS4 values for input.

zw