cpplib: new lexer

[Zack Weinberg]

On Tue, Apr 25, 2000 at 11:48:30PM +0900, Neil Booth wrote:
> Neil Booth wrote:-
> 
> >       if (! (list->tokens[1].flags & CPP_OPEN_PAREN))
> > 	cpp_pedwarn (pfile,
> > 		     "The C standard requires whitespace after #define %s",
> > 		     hp->name);
> > 
> > This is testing for a CPP_OPEN_PAREN (which isn't even a flag) rather
> > than HSPACE_BEFORE.  I quickly checked if that solved the problem, but
> > it didn't, so I reverted the patch.  Looking at it again maybe the
> > index should be 2 not 1?
> 
> The index 1 is fine.  The following patch bootstraps the C compiler,
> and replaces the CPP_OPEN_PAREN with PREV_WHITESPACE above.  I'd
> inserted a dummy token at the front of every list, forgetting you'd
> starting using them Zack.  The dummy token is needed so we don't
> underflow the buffer.  So now I just get list->tokens to point to the
> first "real" token after the dummy token, rather than try to update
> cpphash.c.

I have now bootstrapped and tested this.  Go ahead and put it back.
However, please add commentary to init_token_list and free_token_list
explaining why list->tokens is adjusted in this way.

Also, the changelog entry should read something like

      Restore previous patch with fixes.

      * cpplex.c (init_token_list, free_token_list): Set list->tokens
      to point one entry past the beginning of the list.
      * cpphash.c (_cpp_create_definition): Fix thinko: replace
      CPP_OPEN_PAREN with PREV_WHITESPACE in flags check.

plus any other changes beyond the previous patch.

zw