libcpp how-to question: Tokenizing and spaces & tabs – or special Fortran needs

[Dodji Seketeli]

[I have changed Tom Tromey's email address to his current one]

Hello Tobias,

Just for the record -- as I am trimming the original post for legibility
-- the initial message I am replying to can be read at
https://gcc.gnu.org/ml/gcc/2014-11/msg00357.html.

Tobias Burnus <burnus@net-b.de> writes:

[...]

> Do you have a suggestion how to best implement this white-space
> preserving with libcpp? It can (and presumably should) be a special
> flag/function for Fortran.

I would propose that libcpp gets extended to gain a new kind of token
which type would be something like 'CPP_WHITESPACE', which would contain
the exact spelling of the continuous non-vertical spaces that are
discarded today.

There would then be a new libcpp option that would actually make
cpp_get_token() yield that kind of token.  The rest of the behaviour of
cpp_get_token() that is today associated with white spaces would remain
mostly unchanged.

The Fortran front-end would then initialize the cpp_reader type with the
new option so that cpp_get_token() yields the verbatim non-vertical
white space tokens.

For the case of strings that could have comments in the middle, with
associated continuation lines, I guess we could handle these especially
in the lex_string() function in libcpp/lex.c.  But then the comment
would be dropped on the floor.

Would these changes be useful enough?

Cheers,

-- 
		Dodji