Lexing and multibyte character sets

[Neil Booth]

I wrote a patch to the lexer recently that will allow some much-needed
enhancements in the future.  Zack rejected it, because it does some
re-scanning of the input characters, and this is not good for handling
stateful streams of characters in the future.

So, I'd like to ask what is needed for future handling of these
character sets, so I can get the patch altered and make dropping in
multibyte character support in the future simple.

Am I right that

a) in the future we will have to be able to lex the input file a
wchar_t at a time?  I only mean this in general; I'm sure there will
be a special case for the normal 100% superset of ASCII case.

b) because of a), it is desirable to eliminate any kind of
back-searching or retracing of characters from the lexer?  Zack left
an enigmatic comment at the end of his update to proj-cpplib.html,
saying stepping backwards was possible.

If this is the case, I'd like to make the changes b) now, so that
multibyte support is, theoretically at least, achieved simply by
replacing byte-reading of characters by calls to mbrtowc, iconv or
suchlike?

Or am I being too harsh on myself with the restrictions in b)?  For
example, I expect that since, internally, in the worst case we'll be
handling already-translated wchar_t arrays, stepping backwards through
already-lexed strings and identifiers is not a problem.

Thanks for any advice,

Neil.