Streamtokenizer

[Tom Tromey]

>>>>> "OneGuy" == OneGuy  <oneguyks@gmail.com> writes:

OneGuy> Memory usage was even worse (55 MB with Classpath vs 35 MB with Sun's).

It looks like our nextToken will often create a new StringBuffer.
Perhaps it would be better to make one and cache it.

FWIW I'd accept a clean patch to further optimize StreamTokenizer.
Most likely the only reason it performs poorly is that it never popped
up as a performance limiter before.

Tom