cpplib: new lexer (0/3)

[Zack Weinberg]

After two and a half months of work, cpplib's new lexer and macro
expander are ready to see the light of day.  This patch is in three
chunks: the core, adjustments to fix-header, and new or modified test
cases.

This is a major overhaul, with approximately 8,000 lines of code
touched in the core library alone.  I have tested extensively; in
addition to GCC's own testsuite, it has passed two different C89
conformance suites, and successfully compiled glibc, X11 (but see
below), XEmacs, and the Linux kernel.  Even then, there will
undoubtedly be bugs.  Please do report any you encounter; they will be
fixed faster if you provide a self-contained test case, and even
faster if you provide a patch (hint, hint).

You are likely to encounter problems if you are abusing the
preprocessor to munge something other than C, C++, or Objective C
source code.  Those problems will still be fixed if reported, but they
have less priority than problems with legitimate source code.  You
should try using M4 instead.

K+R compatibility mode is known to have serious bugs.  I am looking
into solutions.  Please do not report -traditional bugs unless you
can provide a complete test case; odds are I already know about them.

Imake is known not to work with this code.  Again, please do not
bother reporting bugs in support for imake; I already know about them.
(But simple self-contained test cases would be appreciated.)

In the default configuration, imake uses /lib/cpp; if you preserve
that as a link to the /usr/bin/cpp installed by 2.95 or an earlier
version, and keep the corresponding /usr/lib/gcc-lib directory around,
you should have no trouble compiling X11.  You can also cause imake to
be a bit more flexible by defining FIXUP_CPP_WHITESPACE in imakemdep.h.
In my tests this cures most but not all of the problems.

The good news is, this code is faster, smaller, and cleaner.  It will
be easier to integrate into the C front end.  It will enable several
further optimizations.  And those obnoxious warnings about trigraphs
in comments are history.

I would like to thank Neil Booth (neilb@earthling.net), who devised
the algorithms and wrote most of the new code.

zw