cpplib: lexer cleanup

[Zack Weinberg]

On Mon, May 08, 2000 at 11:16:10PM +0900, Neil Booth wrote:
> This includes the "lexer touch ups" of the previous thread, fixes a
> whitespace bug over a backslash newline, folds all the spelling
> functions into spell_token, and generally cleans up.  Zack pointed out
> I was mis-parsing assertions; this fixes that too.

Too many changes in the same patch.  Could you split it into:

- the pure bug fixes
- folding all the spelling functions
- the changes to comment handling
- the assertion changes

and send each separately, please?  Also, you should test your patches
*before* you submit them.  For code like this where it's not currently
used, it suffices to make sure it compiles (you needn't run a full
bootstrap).

zw