java.io/313: StreamTokenizer does not handle /* comments correctly

[osk at hem dot passagen dot se]

>Number:         313
>Category:       java.io
>Synopsis:       StreamTokenizer does not handle /* comments correctly
>Confidential:   no
>Severity:       serious
>Priority:       medium
>Responsible:    tromey
>State:          open
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Wed Aug 09 12:16:01 PDT 2000
>Closed-Date:
>Last-Modified:
>Originator:     Oskar Liljeblad
>Release:        recent libgcj (not the latest though)
>Organization:
>Environment:

>Description:
The class java.io.StreamTokenizer doesn't handle '/*'-comments
correctly. This is because the libgcj/Classpath StreamTokenizer
checks for commentChar's first, then slashStar-comments.
The '/' character is by default set as a comment character - 
a comment that ends at EOL.

Oskar
>How-To-Repeat:
import java.io.IOException;
import java.io.StringReader;
import java.io.StreamTokenizer;

public class PRxxx {
	public static void printTokenization(String s) {
		StringReader r = new StringReader(s);
		StreamTokenizer st = new StreamTokenizer(r);
		st.slashStarComments(true);
		try {
			while (st.nextToken() != StreamTokenizer.TT_EOF) {
				System.out.print(st.ttype+"["+st.sval+","+st.nval+"] ");
			}
		} catch (IOException e) {
			System.out.println("[IOException]");
		}
		System.out.println();
	}

	public static void main(String[] args)
	throws IOException {
		printTokenization("alpha /* bleh */ gamma");
		printTokenization("alpha / bleh");
		printTokenization("alpha /* bleh");
	}
}

// Good output (note the extra space at end of lines):
// -3[alpha,0.0] -3[gamma,0.0] 
// -3[alpha,0.0] 
// -3[alpha,0.0] 
>Fix:
See the patch0 file. This patch may introduce other (subtle)
bugs so it would be nice if the original author (Warren Levy)
or someone else could verify it.
>Release-Note:
>Audit-Trail:
>Unformatted:
----gnatsweb-attachment----
Content-Type: application/octet-stream; name="patch0"
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="patch0"

LS0tIFN0cmVhbVRva2VuaXplci5qYXZhLm9sZAlTYXQgQXVnICA1IDA5OjMzOjU0IDIwMDAKKysr
IFN0cmVhbVRva2VuaXplci5qYXZhCVdlZCBBdWcgIDkgMjA6NTI6MjIgMjAwMApAQCAtMjg5LDYg
KzI4OSw1MCBAQAogCSAgICByZXR1cm4gKHR0eXBlID0gVFRfRU9MKTsKIAl9CiAKKyAgICBpZiAo
Y2ggPT0gJy8nKQorICAgICAgaWYgKChjaCA9IGluLnJlYWQoKSkgPT0gJy8nICYmIHNsYXNoU2xh
c2gpCisgICAgCXsKKyAgICAJICB3aGlsZSAoKGNoID0gaW4ucmVhZCgpKSAhPSAnXG4nICYmIGNo
ICE9ICdccicgJiYgY2ggIT0gVFRfRU9GKQorICAgIAkgICAgOworICAgIAkgIGlmIChjaCAhPSBU
VF9FT0YpCisgICAgCSAgICBpbi51bnJlYWQoY2gpOworICAgIAkgIHJldHVybiBuZXh0VG9rZW4o
KTsgLy8gUmVjdXJzaXZlLCBidXQgbm90IHRvbyBkZWVwIGluIG5vcm1hbCBjYXNlcworICAgIAl9
CisgICAgICBlbHNlIGlmIChjaCA9PSAnKicgJiYgc2xhc2hTdGFyKQorICAgIAl7CisgICAgCSAg
d2hpbGUgKHRydWUpCisgICAgCSAgICB7CisgICAgCSAgICAgIGNoID0gaW4ucmVhZCgpOworICAg
IAkgICAgICBpZiAoY2ggPT0gJyonKQorICAgIAkJeworICAgIAkJICBpZiAoKGNoID0gaW4ucmVh
ZCgpKSA9PSAnLycpCisgICAgCQkgICAgYnJlYWs7CisgICAgCQkgIGVsc2UgaWYgKGNoICE9IFRU
X0VPRikKKyAgICAJCSAgICBpbi51bnJlYWQoY2gpOworICAgIAkJfQorICAgIAkgICAgICBlbHNl
IGlmIChjaCA9PSAnXG4nIHx8IGNoID09ICdccicpCisgICAgCQl7CisgICAgCQkgIGxpbmVOdW1i
ZXIrKzsKKyAgICAJCSAgaWYgKGNoID09ICdccicgJiYgKGNoID0gaW4ucmVhZCgpKSAhPSAnXG4n
KQorICAgIAkJICAgIHsKKyAgICAJCSAgICAgIGlmIChjaCAhPSBUVF9FT0YpCisgICAgCQkJaW4u
dW5yZWFkKGNoKTsKKyAgICAJCSAgICB9CisgICAgCQl9CisgICAgCSAgICAgIGVsc2UgaWYgKGNo
ID09IFRUX0VPRikKKyAgICAJCXsKKyAgICAJCSAgYnJlYWs7CisgICAgCQl9CisgICAgCSAgICB9
CisgICAgCSAgcmV0dXJuIG5leHRUb2tlbigpOyAvLyBSZWN1cnNpdmUsIGJ1dCBub3QgdG9vIGRl
ZXAgaW4gbm9ybWFsIGNhc2VzCisgICAgCX0KKyAgICAgIGVsc2UKKyAgICAJeworICAgIAkgIGlm
IChjaCAhPSBUVF9FT0YpCisgICAgCSAgICBpbi51bnJlYWQoY2gpOworICAgIAkgIGNoID0gJy8n
OworICAgIAl9CisKICAgICBpZiAoY2ggPT0gVFRfRU9GKQogICAgICAgdHR5cGUgPSBUVF9FT0Y7
CiAgICAgZWxzZSBpZiAoaXNOdW1lcmljKGNoKSkKQEAgLTQwNSw1MCArNDQ5LDYgQEAKICAgICAg
IH0KICAgICBlbHNlCiAgICAgICB7Ci0gICAgICAgIGlmIChjaCA9PSAnLycpCi0JICBpZiAoKGNo
ID0gaW4ucmVhZCgpKSA9PSAnLycgJiYgc2xhc2hTbGFzaCkKLQkgICAgewotCSAgICAgIHdoaWxl
ICgoY2ggPSBpbi5yZWFkKCkpICE9ICdcbicgJiYgY2ggIT0gJ1xyJyAmJiBjaCAhPSBUVF9FT0Yp
Ci0JCTsKLQkgICAgICBpZiAoY2ggIT0gVFRfRU9GKQotCQlpbi51bnJlYWQoY2gpOwotCSAgICAg
IHJldHVybiBuZXh0VG9rZW4oKTsgLy8gUmVjdXJzaXZlLCBidXQgbm90IHRvbyBkZWVwIGluIG5v
cm1hbCBjYXNlcwotCSAgICB9Ci0JICBlbHNlIGlmIChjaCA9PSAnKicgJiYgc2xhc2hTdGFyKSAK
LQkgICAgewotCSAgICAgIHdoaWxlICh0cnVlKQotCQl7Ci0JICAgICAgICAgIGNoID0gaW4ucmVh
ZCgpOwotCQkgIGlmIChjaCA9PSAnKicpCi0JCSAgICB7Ci0JCSAgICAgIGlmICgoY2ggPSBpbi5y
ZWFkKCkpID09ICcvJykKLQkJCWJyZWFrOwotCQkgICAgICBlbHNlIGlmIChjaCAhPSBUVF9FT0Yp
Ci0JCQlpbi51bnJlYWQoY2gpOwotCQkgICAgfQotCQkgIGVsc2UgaWYgKGNoID09ICdcbicgfHwg
Y2ggPT0gJ1xyJykKLQkJICAgIHsKLQkJICAgICAgbGluZU51bWJlcisrOwotCQkgICAgICBpZiAo
Y2ggPT0gJ1xyJyAmJiAoY2ggPSBpbi5yZWFkKCkpICE9ICdcbicpCi0JCQl7Ci0JCSAgICAgICAg
ICBpZiAoY2ggIT0gVFRfRU9GKQotCQkJICAgIGluLnVucmVhZChjaCk7Ci0JCQl9Ci0JCSAgICB9
Ci0JCSAgZWxzZSBpZiAoY2ggPT0gVFRfRU9GKQotCQkgICAgewotCQkgICAgICBicmVhazsKLQkJ
ICAgIH0KLQkJfQotCSAgICAgIHJldHVybiBuZXh0VG9rZW4oKTsgLy8gUmVjdXJzaXZlLCBidXQg
bm90IHRvbyBkZWVwIGluIG5vcm1hbCBjYXNlcwotCSAgICB9Ci0JICBlbHNlCi0JICAgIHsKLQkg
ICAgICBpZiAoY2ggIT0gVFRfRU9GKQotCQlpbi51bnJlYWQoY2gpOwotCSAgICAgIGNoID0gJy8n
OwotCSAgICB9Ci0KIAl0dHlwZSA9IGNoOwogICAgICAgfQogCg==