Re: First lexer tests, plus small trigraph bugfix

[Neil Booth <NeilB at earthling dot net>]

Per Bothner wrote:-

> If you can't test it, I guess it doesn't matter?

Admittedly probably not.  It would become more important with an
integrated front-end, that requests tokens one at a time.  The problem
I face in devising a test is that behaviour will probably differ
between an integrated front-end that gets 2 tokens from a buggy lexer,
and one that reads the adjacent tokens in from a preprocessed file and
tokenises it into a single one by itself.

The only part of the preprocessor that doesn't treat tokens as blocks
but as individuals is pasting of arguments.  I couldn't think of any
way to exploit that either, though.

> Is there actually any reason we really need to worry about the concept
> of "preprocessing number"?  Why not just have the integrated cpplib and
> front-end-lexer just lex valid numbers?

Two valid preprocessing numbers, invalid as numbers, could paste to
become a single token valid as a number, is an example that
immediately comes to mind.

Hmm, digging up the C9X Rationale statement, I quote:-

6.4.8 Preprocessing Numbers

The notion of preprocessing numbers was introduced to simplify the
description of preprocessing.  It provides a means of talking about
the tokenization of strings that look like numbers, or initial
substrings of numbers, prior to their semantic interpretation.  In the
interests of keeping things simple, occasional spruious forms are
scanned as preprocessing numbers. [...]  It was felt this anomaly was
no worse than the principle under which the characters a+++++b are
tokenized as a ++ ++ +b (an invalid expression) [...]

It goes on to give an example from real code of it making a difference
with pasting, and somewhere else mentions the desire to keep their
options open for future changes.

When we integrate the front-end with cpplib, we will probably want to
move interpretation to an earlier stage.  Also, Zack is wanting to do
this anyway within the existing preprocessor I think.

Neil.