java.io/141: StreamTokenizer doesn't scan mixed token types correctly

[jsturm@sigma6.com]

>Number:         141
>Category:       java.io
>Synopsis:       StreamTokenizer doesn't scan mixed token types correctly
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    tromey
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sun Jan 23 13:30:00 PST 2000
>Closed-Date:
>Last-Modified:
>Originator:     Jeff Sturm
>Release:        libgcj (current)
>Organization:
>Environment:
alphapca56-unknown-linux-gnu
>Description:
While trying to scan the input below:

(a).(b)

output of StreamTokenizer (from libgcj) doesn't match that
of the JDK.
>How-To-Repeat:
Compile/run the attached program:

gcj StreamTokenizerTest.java --main=StreamTokenizerTest -o st
./st '(a).(b)'

Expected output is:

(
[TT_WORD]: a
)
[TT_NUMBER]: 0.0
(
[TT_WORD]: b
)
[TT_EOF]
>Fix:

>Release-Note:
>Audit-Trail:
>Unformatted:
----gnatsweb-attachment----
Content-Type: application/octet-stream; name="StreamTokenizerTest.java"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="StreamTokenizerTest.java"

aW1wb3J0IGphdmEuaW8uSU9FeGNlcHRpb247Ci8vaW1wb3J0IGphdmEuaW8uU3RyZWFtVG9rZW5p
emVyOwppbXBvcnQgamF2YS5pby5TdHJpbmdSZWFkZXI7CgpwdWJsaWMgY2xhc3MgU3RyZWFtVG9r
ZW5pemVyVGVzdCB7CglwdWJsaWMgc3RhdGljIHZvaWQgdGVzdChTdHJpbmcgc3RyKSB0aHJvd3Mg
SU9FeGNlcHRpb24gewoJCVN0cmluZ1JlYWRlciByZWFkZXIgPSBuZXcgU3RyaW5nUmVhZGVyKHN0
cik7CgkJU3RyZWFtVG9rZW5pemVyIHN0ID0gbmV3IFN0cmVhbVRva2VuaXplcihyZWFkZXIpOwoJ
CWludCB0ID0gMDsKCgkJZG8gewoJCQl0ID0gc3QubmV4dFRva2VuKCk7CgoJCQlzd2l0Y2ggKHQp
IHsKCQkJCWNhc2UgU3RyZWFtVG9rZW5pemVyLlRUX1dPUkQ6CgkJCQkJU3lzdGVtLm91dC5wcmlu
dGxuKCJbVFRfV09SRF06ICIgKyBzdC5zdmFsKTsKCQkJCQlicmVhazsKCQkJCWNhc2UgU3RyZWFt
VG9rZW5pemVyLlRUX05VTUJFUjoKCQkJCQlTeXN0ZW0ub3V0LnByaW50bG4oIltUVF9OVU1CRVJd
OiAiICsgc3QubnZhbCk7CgkJCQkJYnJlYWs7CgkJCQljYXNlIFN0cmVhbVRva2VuaXplci5UVF9F
T0w6CgkJCQkJU3lzdGVtLm91dC5wcmludGxuKCJbVFRfRU9MXSIpOwoJCQkJCWJyZWFrOwoJCQkJ
Y2FzZSBTdHJlYW1Ub2tlbml6ZXIuVFRfRU9GOgoJCQkJCVN5c3RlbS5vdXQucHJpbnRsbigiW1RU
X0VPRl0iKTsKCQkJCQlicmVhazsKCQkJCWRlZmF1bHQ6CgkJCQkJU3lzdGVtLm91dC5wcmludGxu
KChjaGFyKXQpOwoJCQkJCWJyZWFrOwoJCQl9CgkJfSB3aGlsZSAodCAhPSBTdHJlYW1Ub2tlbml6
ZXIuVFRfRU9GKTsKCX0KCglwdWJsaWMgc3RhdGljIHZvaWQgbWFpbihTdHJpbmdbXSBhcmdzKSB7
CgkJZm9yIChpbnQgbiA9IDA7IG4gPCBhcmdzLmxlbmd0aDsgbisrKSB7CgkJCXRyeSB7CgkJCQl0
ZXN0KGFyZ3Nbbl0pOwoJCQl9IGNhdGNoIChUaHJvd2FibGUgdCkgewoJCQkJdC5wcmludFN0YWNr
VHJhY2UoKTsKCQkJfQoJCX0KCX0KfQo=