ideas for cpplib

[Zack Weinberg]

On Thu, 08 Oct 1998 15:13:35 -0400, Dave Brolley wrote:
>Zack Weinberg wrote:
>
>> I have some ideas for cpplib, which I would like to run by you.
>>
>> * Profiling indicates an overwhelming amount of the CPU time consumed
>> by cpplib is in adjust_position() and its immediate callers
>> (update_position, cpp_buf_line_and_col). These functions could go
>> away if pfile->buffer->lineno were kept up-to-date in all the places
>> that read from the buffer.  That's pretty easy to do.  The only
>> loss is column information, and that is barely ever used.
>>
>> A trial implementation didn't speed up noticeably.  This is because
>> cpplib is bound by directory search and I/O in my usual test
>> (compiling glibc).  A less disk-intensive compile might benefit more.
>> There's a big win on code comprehensibility.
>
>My take on stuff like this is that the most important test cases are real code
>(such as glibc). If you think that the readability of the code would
>be improved significantly, then I would be happy to look at a patch.

I had a patch but I think it wasn't doing the right thing always, and
I want to get the bugfixes done first.

>> * -dM is not implemented.  It seems to me that the tidiest way to
>> implement it is to move macro and directive expansion out of
>> cpp_get_token into its caller.  That allows the caller to control the
>> process.  -dM can then be implemented by skipping everything but
>> directive lines, executing only #define and #undef, and dumping the
>> hash table at the end.  (I think conditional_skip could get faster
>> too.)
>
>The way we did this in another compiler I worked on was to write a second
>scanner which did only the minimum needed to maintain context (strings,
>comments, and a few other things). This scanner called the same directive
>handler as the main scanner, but was significantly faster. I think we
>also used this second scanner to do the conditional skipping.

This is essentially what I had in mind.  Both scanner loops would call
the same lexer (cpp_get_token) but they would choose to process stuff
or not.

>> * For ANSI conformance we need trigraphs.  The obvious place to put
>> them is in GETC() but that would entail a test on every character
>> input, which would be horrible performance-wise.  What do you think
>> about a prescan in safe_file_read or thereabouts, which would do
>> trigraph replacement (if active), backslash-newline (inserting #line
>> as necessary) and comment deletion?  Much hair then disappears from
>> cpp_get_token.
>
>cccp uses a pre scan to do trigraphs, so the idea is certainly viable.
>
>One problem with doing backslash-newline in a pre-scan is that you lose the
>information you need to get line numbers correct. I see that you want to solve
>this by adding #line directives, but what about backslash-newlines that appear
>on other directives? You can't simply add a #line after each backslah-newline
>you replace. So now you're parsing directives in order to corretly place the
>generated #line directives. I'm not sure that it's a win.

I have a partial implementation which simply defers the newlines until
after the first non-backslashed newline.  In other words,

#define xyz \
a \
b \
c
thingy

becomes

#define xyz a b c



thingy

and the same for comments.  This generates syntactically correct
output for all the testcases I can find.  A problem is that line
number information will be wrong inside a complex multiline macro -
but right now it's wrong anyway (you get errors on the line that
called the macro, not inside the macro text).

It turns out to be a huge memory win, because I don't read the file
all at once, I use stdio and enlarge the intermediate buffer as
necessary.  combine.c is 404K half of which is comments.  The current
cpp allocates a buffer the size of the file; my code needs only 252K.

Right now there are nasty issues with trigraphs and backslashes:
'??/??/' needs to become '\\', and not escape the closing quote, but
??/??/<newline> is a \ followed by an escaped newline.  If I cut out
the trigraph code, the rest works.


>Comments which span lines cause similar problems. Also you need to be
>able to emit comments if -C is specified, so most of that "hair"
>would have to remain in cpp_get_token.

The comment hair isn't nearly as bad as the backslash-newline hair.
Besides, some of it has to stay anyway, because of -traditional.

>> * If we translate #pragma xyz to _Pragma("xyz"); always, then a
>> special case can be removed from c-lex.c.  (Of course, _Pragma needs
>> to be implemented first.)
>>
>> zw
>
>What about #pragmas that appear in arbitrary places?
>
>int i =
>#pragma foo
>1;
>
>Translating all #pragmas to _Pragma would complicate things for the parser.

That's legal?! Yuck.  I thought all the defined pragmas only make
sense as separate entities, not in the middle of statements.

It doesn't clean up c-lex as much as I thought, because of #line.  Oh
well. 

zw