Patches for java.io.StreamTokenizer

[Jeff Sturm <jsturm at sigma6 dot com>]

The patch below prevents an infinite loop in StreamTokenizer.  (Attempting
to unread(-1) on a PushbackReader causes the next read to return 65535.  I
thought this was a bug in PushbackReader, but other implementations seem
to do the same thing.  I guess the consequence of unread(-1) is simply
undefined.)

There are other problems in StreamTokenizer that probably can't be fixed
by a small patch.  For example, "-." is incorrectly read as a number,
causing NumberFormatException.  IMHO the body of nextToken() needs to be
reimplemented as a FSM.  Would it be worthwhile to rewrite this class, or
can we consider getting it from the Classpath project instead?


2000-01-15  Jeff Sturm	<jsturm@sigma6.com>

	* java/io/StreamTokenizer.java (nextToken): Avoid unread(TT_EOF).

Index: StreamTokenizer.java
===================================================================
RCS file: /cvs/java/libgcj/libjava/java/io/StreamTokenizer.java,v
retrieving revision 1.1.1.1
diff -u -r1.1.1.1 StreamTokenizer.java
--- StreamTokenizer.java	1999/04/07 14:52:36	1.1.1.1
+++ StreamTokenizer.java	2000/01/15 15:38:03
@@ -179,7 +179,7 @@
 
 	  // Throw away \n if in combination with \r.
 	  if (ch == '\r' && (ch = in.read()) != '\n')
-	    in.unread(ch);
+	    if (ch != TT_EOF) in.unread(ch);
 	  if (eolSignificant)
 	    return (ttype = TT_EOL);
 	}
@@ -192,7 +192,7 @@
 	  {
 	    // Read ahead to see if this is an ordinary '-' rather than numeric.
 	    ch = in.read();
-	    in.unread(ch);
+	    if (ch != TT_EOF) in.unread(ch);
 	    if (isNumeric(ch) && ch != '-')
 	      ch = '-';
 	    else
@@ -209,7 +209,7 @@
 	  else
 	    tokbuf.append((char) ch);
 
-	in.unread(ch);
+	if (ch != TT_EOF) in.unread(ch);
 	ttype = TT_NUMBER;
 	nval = Double.valueOf(tokbuf.toString()).doubleValue();
       }
@@ -219,7 +219,7 @@
 	tokbuf.append((char) ch);
 	while (isAlphabetic(ch = in.read()) || isNumeric(ch))
 	  tokbuf.append((char) ch);
-	in.unread(ch);
+	if (ch != TT_EOF) in.unread(ch);
 	ttype = TT_WORD;
 	sval = tokbuf.toString();
 	if (lowerCase)
@@ -229,7 +229,7 @@
       {
 	while ((ch = in.read()) != '\n' && ch != '\r' && ch != TT_EOF)
 	  ;
-	in.unread(ch);
+	if (ch != TT_EOF) in.unread(ch);
 	return nextToken();	// Recursive, but not too deep in normal cases.
       }
     else if (isQuote(ch))
@@ -277,7 +277,7 @@
 			  }
 		      }
 
-		    in.unread(nextch);
+		    if (nextch != TT_EOF) in.unread(nextch);
 		}
 
 	    tokbuf.append((char) ch);
@@ -285,7 +285,7 @@
 
 	// Throw away matching quote char.
 	if (ch != ttype)
-	  in.unread(ch);
+	  if (ch != TT_EOF) in.unread(ch);
 
 	sval = tokbuf.toString();
       }
@@ -296,7 +296,7 @@
 	    {
 	      while ((ch = in.read()) != '\n' && ch != '\r' && ch != TT_EOF)
 		;
-	      in.unread(ch);
+	      if (ch != TT_EOF) in.unread(ch);
 	      return nextToken(); // Recursive, but not too deep in normal cases
 	    }
 	  else if (ch == '*' && slashStar) 
@@ -308,16 +308,15 @@
 		    if ((ch = in.read()) == '/')
 		      break;
 		    else
-		      in.unread(ch);
+		      if (ch != TT_EOF) in.unread(ch);
 		  else if (ch == '\n' || ch == '\r')
 		    {
 		      lineNumber++;
 		      if (ch == '\r' && (ch = in.read()) != '\n')
-			in.unread(ch);
+		        if (ch != TT_EOF) in.unread(ch);
 		    }
 		  else if (ch == TT_EOF)
 		    {
-		      in.unread(ch);
 		      break;
 		    }
 		}
@@ -325,7 +324,7 @@
 	    }
 	  else
 	    {
-	      in.unread(ch);
+	      if (ch != TT_EOF) in.unread(ch);
 	      ch = '/';
 	    }
 
--
Jeff Sturm
jsturm@sigma6.com