-traditional comment elimination (yuk...)

[Neil Booth <NeilB at earthling dot net>]

Zack,

A quickie about exactly how comment removal works -traditionally.  I
think I've discovered an obscure bug in the preprocessor,
(interestingly cccp seems to be OK) and we only get away with it
because the front end does it's own lexing.  It also reveals a bug in
my current lexer.  I realised this when thinking about handling
removal of escaped newlines within parse_number and parse_string.

Input        New Lexer    Current CPP    Current Front End

&/**/&          <&&>         <&><&>           <&&>
foo/**/bar   <foo><bar>    <foo><bar>       <foobar>
123/**/.456  <123><.456>   <123><.456>      <123.456>

My understanding of traditional comments is that the current CPP is
wrong for all lines, and I'm wrong for the last 2 because I don't do
identifier and number recombination with themselves, *except for the
escaped newline case*.  Am I right in my understanding of the correct
tokenisations?

This persuades me that keeping escaped-newline recombination outside
of parse_number and parse_name is probably a good idea, as there will
be another user; namely -traditional after a C-style comment.

The bug is easily obtainable, even for operators, within the current
preprocessor:-

#if 1 &/**/& 1
#endif

bash-2.03$ ./cpp ~/test.c -traditional
# 1 "/home/neil/test.c"
/home/neil/test.c:1:12: operator '&' has no right operand

I'll add a testcase for this when I get time.  This is all very ugly.
I dream of the day we dump -traditional.

Neil.