RE: C++ lexer (GCC 3.1.1) requires knowledge of other C dialects

["Gary Funck" <gary at Intrepid dot Com>]

Stan Shebs wrote:

> Yacc-class generators turn out to be fundamentally inadequate for
> C++, and in retrospect it looks like it wasn't such a great idea to
> produce a C++ compiler as a completely separate frontend from
> the C compiler - over the years there has had to be a bunch of
> duplicated work, duplicated bug reports, etc.  (Could Tiemann
> have been sufficiently clairvoyant?  Perhaps not...)
> 

An LR-based compiler generator like YACC does have its difficulties when
confronted by the realities of a rather quirky language like C (and C++).
It may be the case that perhaps too much emphasis was placed upon a completely
parser-directed semantic analysis and that more analysis should have been
off-loaded to additional special purpose parsing code (ala grokdeclarator).
Similarly a top-down recursive descent parser will be challenged. My point
is however that hand-built parsers are notoriously difficult to build in a
way that they parse the intended language, and only that language. More
importantly, these parsers can be difficult to extend reliably. Therefore,
using either parsing approach, I'd suggest that the resulting parser will
be easier to maintain and extend if it is built using a parser-generator
or related support/analysis tools.

> To be honest, support for new dialects of C has not really been a
> design requirement for the GCC frontends, and I think it would be
> impractical to add such a requirement, too vague to specify. 

Perhaps, but as above, it will likely be easier to extend the parser
if it is built using some degree of automation, and if there is some
analysis capability that insures that the resulting parser recognizes
the intended language.

Although C seems like a stable language, so far we've seen K&R C, ANSI C,
C++, Objective-C, C99, and perhaps soon C# (perish the thought)? It seems to
me that planning for new dialects should be an important design goal.

> From my one bit of experience (AltiVec extensions to C and C++), yacc
> wasn't really a help anyway, because the extensions had to be context
> sensitive in various ways, and the parser changes proper were the
> smallest part of the support code.

Yes, in many cases the parser extensions are minimal, but one reason for
that is that you can rely upon the parser generator tool (yacc/bison in
GCC's case) to do the heavy lifting - to build a new recognizer and to
complain about ambiguities in the grammar specification.

PS: I did take a look at Mark's recursive descent parser. It's well-structured
and well thought-out. My comments above are intended only to point out some
of the difficulties that might arise when trying to extend the parser, and
to state a prefered approach (ie, to use some amount of automation).