Re: [patch] new exec-charset testcase

[Mark Mitchell <mark at codesourcery dot com>]

Eric Christopher wrote:

> > > Any other thoughts?
> >
> > Well, pushing cpp_tokens during a tentative parse would be a good
> > thing, I think, it would reduce backtracking overhead.  I don't have
> > any idea how hard it would be though.  Mark, can you comment?
>
>
> This is my preference too :)

I'm not sure what you're asking.  I think you're asking "can we save 
cpp_tokens instead of cp_tokens?"  The answer is definitely negative. 
There are situations where we must save the trees that we have generated 
 for both correctness and efficiency reasons.  A good example is 
template-ids:

  S<int>

We in fact collapse that set of tokens into a single pseudo-token after 
seeing it the first time, remembering the associate *_TYPE node.  That's 
important because if the type was bogus we only want to complain once. 
And, it's a big speed optimization; it avoids doing lookup again.  I 
measureed this and it was a big win.

In the particular case of string constants you could probably have 
cp_lexer_rollback_tokens mark any STRING_CSTs rolled back as dirty, and 
recompute their values when you encounter them again.  But, you don't 
actually want to do that.  It will be expensive.  If you do that, do it 
only in the case where the execution character set is not the same as 
the host character set; we shouldn't penalize the typical case.  A 
better approach would probably be to give the lexer enough smarts to 
know when it had to do the translation.

--
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com