Re: gcc reports Internal Error

[Neil Booth <neil at daikokuya dot co dot uk>]

Zack Weinberg wrote:-

> > Interesting. Perhaps this shouldn't be the case when reading stdin. It
> > would be nice to compile what's available instead of waiting for a
> > long time for all of the code to come through the pipe, then
> > compiling. Parallelism vs serialisation.
> 
> It's done that way to simplify the lexer implementation.  Whether or
> not the lexer is actually simpler is an open question; as I said, it
> may well change in the future.

I've often wondered whether it might be worth reading chunks at a time,
or at least supporting this behaviour.

Suppose we read in a chunk of size N.  If we can find the last newline,
we can put a NUL after it (remembering what was there originally).
If we can't find a newline, keep reading chunks until we do.  Because
of the range of newlines we find, it's best to actually replace the first
newline char in the last string of newline chars, if you see what I
mean.

Then, just change the handle_nul routine in cpplex.c (and something
similar in cpptrad.c) to copy the tail bytes of the previous block
to the start of the buffer, read a new block in after them, and repeat.
If really at EOF, continue as before.

The worst case is a file with no newlines; in that case we just read the
whole lot in like we do now.

Neil.