The behavior of java.io.StreamTokenizer differs from Sun's implementation;
StreamTokenizer can be told about a comment character, but libgcj's
implementation seems to ignore the comment characters sometimes.

Consider the following program:

// Instances.java
// pared-down version of weka.core.Instances from weka-3-4

import java.io.InputStreamReader;
import java.io.StreamTokenizer;

class Instances
{
  public static void main(String[] args)
    throws Exception
  {
    StreamTokenizer tok = new
      StreamTokenizer(new InputStreamReader(System.in));
    initTokenizer(tok);

    for (int t = tok.nextToken()
	; t != tok.TT_EOF
	; t = tok.nextToken()) {
      System.out.println(tok.toString());
    }
  }

  private static void initTokenizer(StreamTokenizer tokenizer) {
    tokenizer.resetSyntax();         
    tokenizer.whitespaceChars(0, ' ');    
    tokenizer.wordChars(' '+1, '\u00FF');
    tokenizer.whitespaceChars(',', ',');
    tokenizer.commentChar('%');
    tokenizer.eolIsSignificant(true);
  }
}

// eof

When this program is linked against Sun's class libraries (tested with J2SDK
1.4.x) the following happens:

$ java Instances
  %foo,bar baz
Token[EOL], line 2
$ 

But when compiled with gcj and linked against libgcj4, the supplied commentChar
'%' gets ignored:

$ gcj -g -o instances --main=Instances Instances.java
$ ./instances
  %foo,bar baz
Token[%foo], line 1
Token[bar], line 1
Token[baz], line 1
Token[EOL], line 2
$ 

This behavior seems wrong.