cpplib: new lexer

[Neil Booth]

Zack Weinberg wrote:-

> > > Has this patch been bootstrapped? 
> > > 
> > > It breaks my bootstrap on x86-linux giving the error
> > > 
> > > 	"internal error: object-like macro went through full #define"
> 
> ... I'm not sure how that happened.

Well, many things were off by one because of the extra token.  My code
initiated the list to tokens_used equals 1.  Command line -DIN_GCC
created a token list "IN_GCC" "1" "\n".  Because you start at the
beginning of the list, and rely on the token count, you were seeing a
4-token list with "EOF" at the front.  Anything could happen in those
circumstances I guess.

Incidentally, any reason why command line defines are #defined to 1
and not empty?  Not that it really matters.

Neil.