Streamtokenizer

Fri May 9 15:03:00 GMT 2008

On Fri, May 9, 2008 at 5:51 AM, Andrew Haley <aph@redhat.com> wrote:

> Could be.  That's one of the problems of microbenchmarks: they do
> unrealistic things and tend to magnify the importance of a tiny part
> of the code base.  The famous Dhrystone benchmark was hyper-sensitive
> to the performance of string copying.

I am not sure that problem with StreamTokinzer is just limited to
parsing numbers.
I downloaded GNU Classpath StreamTokenizer from the web site so I
could use it with HotSpot VM .

Here is simple program to a count words in a text file (a 40 MB text
file with no numbers).

//import java.io.StreamTokenizer; (commented the import so I can use Classpath)
public class wc  {
   public static void main(String[] args) throws Exception{
      int count = 0;
      StreamTokenizer lineTokenizer = new StreamTokenizer(System.in);
       while (lineTokenizer.nextToken() != StreamTokenizer.TT_EOF) {
       	if (lineTokenizer.ttype == StreamTokenizer.TT_WORD)
       		count++;
       }
       System.out.println(count);
   }
}

$ time java wc <books.txt
24503500

time:   0m8.667s

Now  using the Sun's version of StreamTokenizer

$ time java wc <books.txt
24503500

 time 0m2.502s

Memory usage was even worse (55 MB with Classpath vs 35 MB with Sun's).

If I use, -verbose:gc  with Sun, I get

[GC 3072K->136K(32768K), 0.0023698 secs]
[GC 3208K->144K(35840K), 0.0007296 secs]
[GC 6288K->144K(35840K), 0.0007489 secs]
[GC 6288K->144K(41984K), 0.0007232 secs]
[GC 12432K->144K(41984K), 0.0008445 secs]
[GC 12432K->144K(53952K), 0.0007658 secs]
[GC 24720K->132K(52736K), 0.0008623 secs]
[GC 23620K->124K(51712K), 0.0007501 secs]
[GC 22588K->124K(50944K), 0.0006523 secs]
[GC 21628K->140K(49856K), 0.0019191 secs]
[GC 20748K->140K(49152K), 0.0011001 secs]
[GC 19852K->124K(48128K), 0.0006544 secs]
[GC 19004K->124K(47552K), 0.0009345 secs]
[GC 18236K->140K(46592K), 0.0006815 secs]
[GC 17484K->124K(46080K), 0.0006656 secs]
[GC 16764K->140K(45248K), 0.0007648 secs]
[GC 16140K->124K(44800K), 0.0007418 secs]
[GC 15484K->140K(44032K), 0.0007486 secs]
[GC 14924K->124K(43584K), 0.0008066 secs]
[GC 14332K->140K(42880K), 0.0014719 secs]
[GC 13772K->140K(42496K), 0.0007442 secs]
[GC 13260K->124K(41856K), 0.0006794 secs]
[GC 12732K->124K(41536K), 0.0006694 secs]

If I Use,  -verbose:gc  with Classpath version, I get the following.
Clearly there is a problem StreamTokenizer. It should do much better.
books.txt has no numbers.

[GC 3072K->144K(32768K), 0.0022625 secs]
[GC 3216K->152K(35840K), 0.0007443 secs]
[GC 6296K->168K(35840K), 0.0007809 secs]
[GC 6312K->152K(41984K), 0.0006776 secs]
[GC 12440K->168K(41984K), 0.0009253 secs]
[GC 12456K->168K(53952K), 0.0008036 secs]
[GC 24744K->152K(53952K), 0.0008192 secs]
[GC 24728K->168K(78592K), 0.0007006 secs]
[GC 49320K->148K(76096K), 0.0007517 secs]
[GC 46996K->164K(73920K), 0.0006294 secs]
[GC 44836K->164K(72064K), 0.0005470 secs]
[GC 42788K->164K(69888K), 0.0008353 secs]
[GC 40804K->132K(68224K), 0.0008284 secs]
[GC 38916K->148K(66240K), 0.0008062 secs]
[GC 37140K->164K(64768K), 0.0007201 secs]
[GC 35492K->148K(62976K), 0.0007356 secs]
[GC 33876K->164K(61632K), 0.0007743 secs]
[GC 32356K->164K(60032K), 0.0007642 secs]
[GC 30948K->148K(58880K), 0.0007080 secs]
[GC 29588K->148K(57408K), 0.0007195 secs]
[GC 28308K->164K(56320K), 0.0006516 secs]
[GC 27108K->164K(55040K), 0.0007758 secs]
[GC 25956K->164K(54016K), 0.0008697 secs]
[GC 24804K->148K(52800K), 0.0008416 secs]
[GC 23700K->148K(51904K), 0.0018387 secs]
[GC 22676K->164K(50816K), 0.0007591 secs]
[GC 21732K->164K(50048K), 0.0008642 secs]
[GC 20836K->164K(49024K), 0.0006899 secs]
[GC 19940K->148K(48256K), 0.0006861 secs]
[GC 19092K->164K(47424K), 0.0008549 secs]
[GC 18340K->164K(46720K), 0.0008144 secs]
[GC 17572K->148K(45952K), 0.0009243 secs]
[GC 16852K->148K(45312K), 0.0007187 secs]
[GC 16148K->148K(44608K), 0.0007639 secs]
[GC 15508K->164K(44096K), 0.0007590 secs]
[GC 14948K->164K(43456K), 0.0005775 secs]
[GC 14372K->148K(42944K), 0.0008866 secs]
[GC 13780K->164K(42368K), 0.0007366 secs]
[GC 13284K->148K(41920K), 0.0006999 secs]
[GC 12756K->148K(45056K), 0.0007714 secs]
[GC 15956K->164K(44480K), 0.0005536 secs]
[GC 15332K->164K(43840K), 0.0007348 secs]
[GC 14756K->148K(43328K), 0.0007709 secs]
[GC 14164K->148K(42752K), 0.0007503 secs]
[GC 13652K->148K(45760K), 0.0008349 secs]
[GC 16660K->164K(49408K), 0.0006727 secs]
[GC 20324K->164K(53888K), 0.0008633 secs]
[GC 24804K->148K(52800K), 0.0006140 secs]
[GC 23700K->164K(51776K), 0.0008563 secs]
[GC 22692K->164K(50816K), 0.0007437 secs]
[GC 21732K->164K(49920K), 0.0008409 secs]
[GC 20836K->164K(49024K), 0.0007014 secs]
[GC 19940K->132K(48192K), 0.0007116 secs]
[GC 19076K->164K(47424K), 0.0007528 secs]
[GC 18340K->148K(46656K), 0.0006948 secs]
[GC 17556K->164K(45952K), 0.0007355 secs]
[GC 16868K->132K(45248K), 0.0006863 secs]
[GC 16132K->148K(44608K), 0.0007230 secs]
[GC 15508K->164K(44032K), 0.0007152 secs]
[GC 14948K->164K(43456K), 0.0008106 secs]
[GC 14372K->148K(46336K), 0.0007169 secs]
[GC 17236K->164K(45632K), 0.0007162 secs]
[GC 16548K->164K(44992K), 0.0007300 secs]
[GC 15908K->164K(44352K), 0.0006661 secs]
[GC 15268K->164K(43776K), 0.0007292 secs]
[GC 14692K->148K(43200K), 0.0007528 secs]
[GC 14100K->148K(46016K), 0.0006938 secs]
[GC 16916K->148K(49408K), 0.0007830 secs]
[GC 20308K->148K(48576K), 0.0007734 secs]
[GC 19476K->148K(47744K), 0.0007191 secs]
[GC 18644K->164K(46976K), 0.0007837 secs]
[GC 17892K->164K(46272K), 0.0006908 secs]
[GC 17188K->132K(45568K), 0.0008964 secs]
[GC 16452K->148K(44928K), 0.0007666 secs]
[GC 15828K->164K(48064K), 0.0007926 secs]
[GC 18980K->164K(47296K), 0.0004832 secs]
[GC 18212K->164K(46528K), 0.0006826 secs]
[GC 17444K->148K(45824K), 0.0006882 secs]
[GC 16724K->164K(45184K), 0.0006959 secs]
[GC 16100K->148K(44544K), 0.0006962 secs]
[GC 15444K->148K(47616K), 0.0009187 secs]
[GC 18516K->148K(46848K), 0.0007162 secs]
[GC 17748K->164K(46144K), 0.0006380 secs]
[GC 17060K->148K(45440K), 0.0008634 secs]
[GC 16340K->148K(48704K), 0.0009382 secs]
[GC 19604K->164K(47872K), 0.0007302 secs]
[GC 18788K->148K(47104K), 0.0008619 secs]
[GC 18004K->148K(46400K), 0.0007370 secs]
[GC 17300K->164K(45696K), 0.0006671 secs]
[GC 16612K->148K(45056K), 0.0008416 secs]
[GC 15956K->148K(44416K), 0.0007216 secs]
[GC 15316K->164K(43840K), 0.0007206 secs]
[GC 14756K->148K(43264K), 0.0006327 secs]
[GC 14164K->148K(46080K), 0.0007517 secs]
[GC 16980K->148K(45376K), 0.0006705 secs]
[GC 16276K->148K(44736K), 0.0005730 secs]
[GC 15636K->148K(44096K), 0.0004457 secs]
[GC 14996K->164K(43520K), 0.0007440 secs]
[GC 14436K->164K(42944K), 0.0007228 secs]
[GC 13860K->132K(42432K), 0.0006942 secs]
[GC 13316K->148K(45120K), 0.0007219 secs]
[GC 16020K->148K(44480K), 0.0007216 secs]
[GC 15380K->148K(47552K), 0.0007280 secs]
[GC 18452K->164K(46784K), 0.0003836 secs]
[GC 17700K->164K(46080K), 0.0006690 secs]
[GC 16996K->148K(45376K), 0.0008866 secs]
[GC 16276K->132K(44736K), 0.0007031 secs]
[GC 15620K->164K(44096K), 0.0007164 secs]