Re: -traditional comment elimination (yuk...)

[Neil Booth <NeilB at earthling dot net>]

Hi Zack,

Zack Weinberg wrote:-

> Having read the entire discussion, I think we're working much too
> hard.

Hmm, no, I think you've minunderstood the issue.  See Dave's first
post about the real effect of comments on K&R compilers.

> If I understand correctly, the problems only come up when -traditional
> and -C are used simultaneously.  -traditional by itself shouldn't be a
> problem, as the comments get no tokens and don't set PREV_WHITESPACE
> on the next token, so paste happens normally.  Correct?

The problem is with -traditional period.  As I said in the first mail
in this thread (when I didn't understand traditional comments fully)
names and identifiers don't undergo pasting.

> Also, the problems only come up on directive lines.  -C might prevent
> two tokens from being pasted elsewhere, but that isn't a problem,
> because -C output can't be fed into the compiler anyway; it's for
> human consumption, and a program that reads -C mode will just have to
> deal.  Also correct?

No, the problems will appear on any line, but *only when we combine
cpplib and the front end*.  We get away with it until then because the
front end does its own lexing pass.

> Therefore, I see no particular reason why we can't just shut off -C
> when processing directives other than #define.  That means the
> comments disappear, but the rest of the line's going to disappear too,
> so the comments are less than helpful.

My feelings are moving towards shutting off -C for preprocessing
directives too.

> We still arguably have a problem with something like
> 
> #def/**/ine foo bar

In fact we actually get that right at present, by luck rather than
design.

> in -C -traditional mode only.  But look at this:
> 
> % cat test.c
> #def/**/ine foo bar
> foo
> % /lib/cpp test.c
> # 1 "test.c" 
> test.c: 1: undefined control

But that *is* a K&R compiler's behaviour.

My understand from Dave's post is that with -traditional, the effect
of comments on the preprocessor and front end are different, because
K&R compilers had a (conceptually, at least) separate preprocessor.

To the preprocessor, a traditional comment acts as a separator, just
like PREV_WHITESPACE does for us now.  The difference is that no space
is actually piped into the K&R front end, which does it's own lexing.
Thus the front end sees pasted tokens, and the preprocessor sees
separate tokens.  Right Dave?

In the light of the above, if my understanding is correct, I am now
convinced of the right way forward, and it won't affect the current
lexer too much.

1) Just like a true K&R compiler, we make -traditional go through two
passes, via a temporary file just like Dave suggested.  This means we
get the same tokenisation as a true K&R compiler would.

2) We need an extra flag TRAD_COMMENT for tokens to indicate a
traditional comment, to get the effect of not outputting whitespace.

These changes should be clean and quite small.

I also now intend to move the escaped-newline removal into parse_name
and parse_number, which you prefer Zack.  I prefer it too in the light
of -traditional's meaning.

I also will get rid of the first 2 tokens of directives.

I hope to post incremental patches to achieve all this this weekend.

Neil.