[Bug libgcj/13062] New: StreamTokenizer ignores commentChar

[jansche at cs dot columbia dot edu]

The behavior of java.io.StreamTokenizer differs from Sun's implementation;
StreamTokenizer can be told about a comment character, but libgcj's
implementation seems to ignore the comment characters sometimes.

Consider the following program:

// Instances.java
// pared-down version of weka.core.Instances from weka-3-4

import java.io.InputStreamReader;
import java.io.StreamTokenizer;

class Instances
{
  public static void main(String[] args)
    throws Exception
  {
    StreamTokenizer tok = new
      StreamTokenizer(new InputStreamReader(System.in));
    initTokenizer(tok);

    for (int t = tok.nextToken()
	; t != tok.TT_EOF
	; t = tok.nextToken()) {
      System.out.println(tok.toString());
    }
  }

  private static void initTokenizer(StreamTokenizer tokenizer) {
    tokenizer.resetSyntax();         
    tokenizer.whitespaceChars(0, ' ');    
    tokenizer.wordChars(' '+1, '\u00FF');
    tokenizer.whitespaceChars(',', ',');
    tokenizer.commentChar('%');
    tokenizer.eolIsSignificant(true);
  }
}

// eof

When this program is linked against Sun's class libraries (tested with J2SDK
1.4.x) the following happens:

$ java Instances
  %foo,bar baz
Token[EOL], line 2
$ 

But when compiled with gcj and linked against libgcj4, the supplied commentChar
'%' gets ignored:

$ gcj -g -o instances --main=Instances Instances.java
$ ./instances
  %foo,bar baz
Token[%foo], line 1
Token[bar], line 1
Token[baz], line 1
Token[EOL], line 2
$ 

This behavior seems wrong.

-- 
           Summary: StreamTokenizer ignores commentChar
           Product: gcc
           Version: 3.3.2
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: libgcj
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: jansche at cs dot columbia dot edu
                CC: gcc-bugs at gcc dot gnu dot org
  GCC host triplet: i686-pc-linux-gnu (from config.guess)
GCC target triplet: i486-linux (from gcj -dumpmachine)


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=13062