Language-independent functions-as-trees representation

[Neil Booth]

Richard Kenner wrote:-

>     Figuring out line numbers is painful if all you have is a character
>     offset into the preprocessed source, or something similar.
> 
> Why?  You make a map from line number to starting character number and
> do a binary search.

And that array can get absolutely huge, is always being resized, and
adding to it is a very frequent operation.  At which point it is far
from obvious that you're gaining anything.  I imagine the average
compilation unit in GCC touches 10000 entries here frequently;
Fergus was reporting files he compiles with almost a million lines.

The line maps that CPP uses are pretty minimal in comparison (at a
guess, one hundred for an average compilation unit; for every
enter / leave of a file).  It just requires you keep track of column
(2 bytes) for whatever things you care about.  Often that is just the
tokens of the current logical line, and things that stick around like
DECLs.

Without the column info, cpp's token would have 2 bytes of padding, so
it doesn't even cost anything.

Neil.