Patch to improve cpplib's C++ support

[Zack Weinberg]

On 12 Jul 1999 00:54:06 -0700, Jason Merrill wrote:
>>>>>> Zack Weinberg <zack@rabi.columbia.edu> writes:
>
> > However, I'm working on a revised API to cpplib that is going to behave
> > differently in both the affected areas.  The big difference is that you
> > make one function call and get handed an entire line split up into
> > tokens.  You then can walk through it at your leisure.
>
>Why a line?  The current API of getting one token at a time makes sense to
>me.  I actually find it inconvenient that #-directives are treated as one
>big token rather than a series of tokens like everything else; it means
>that I can't always rely on cpplib pre-tokenization in yylex().

You don't have to know about this in the front-end unless you want to. 
Think of it as a buffering mechanism to avoid having to call cpp_get_token
for every single token in the program - the overhead is substantial.

#-directives appearing in the output will be split into tokens in the near
future.  But I want to stop using these at least for line number
information; why bother when the front end can get this directly from the
cpp_reader structure (or the cpp_token structure in the revised API)?

> > If you stack another input buffer in the middle of parsing the list,
> > it'll still be there when you come back (this is needed anyway for macro
> > expansion).  So cpp_unget becomes a pointer decrement in the front end.
> > On the flip side, you can't back up by part of a token.  cpp_unget as it
> > was in your patch would have let you do that.
>
>No problem; it was only actually used for full tokens.

Good.

Another thing: white space doesn't generate tokens in the new scheme.  Is
that likely to cause problems?

> > As for redirected_input_p, the plan is to make caller pop the buffer all
> > the time (except for macro buffers).
>
>Huh?  Why should the frontend handle #includes?

Given that you need the ability to pop the buffer yourself sometimes, I
think it's cleaner to ask you to do it all the time.  Compare "if you
get a POP token, call cpp_pop_buffer before calling cpp_get_token again" to
"if you get a POP token, you may or may not be expected to call
cpp_pop_buffer, depending on a magic flag whose name doesn't sound like it
has to do with buffer popping".

zw