[Bug testsuite/30119] libjava testsuite output is erratic and unhelpful

[amylaar at gcc dot gnu dot org]

------- Comment #10 from amylaar at gcc dot gnu dot org  2006-12-19 12:57 -------
(In reply to comment #9)
> OK, so it looks like this is not a testsuite problem: the problem really is
> with the program being tested.

It is also a testsuite problem when the tests are randonly passing or failing;
such tests are not suitable as such in an automated testsuite.
I suppose you could write a wrapper to run them several times to get a
reasonable high confidence that the test generally passes - if the test does
not take too long - and maybe the probabilities can also be enhanced by
forking specifically designed other programs to perturb the load, but
unless/ until such infrastructure is put in place, these randomly failing
tests make it effectively impossible to pass a regression test.

I've redone to tests on my baseline where the test passed during the automated
test, and the results are again random:

amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$
/home/amylaar/bld/2006-11-27/i686/gcc/gcj
-B/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/
-B/home/amylaar/bld/2006-11-27/i686/gcc/ --encoding=UTF-8 -C
-I/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite/../libgcj-4.3.0.jar
-g
/home/amylaar/bld/2006-11-27/srcw/libjava/testsuite/libjava.lang/PR18699.java
-d /home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ export
ld_library_path=.:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27/i686/gcc
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ export
LD_LIBRARY_PATH=.:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27/i686/gcc:.:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27/i686/gcc:.:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27/i686/gcc:/home/amylaar/bld/2006-11-27/i686/./bfd/.libs:/home/amylaar/bld/2006-11-27/i686/./prev-bfd/.libs:/home/amylaar/bld/2006-11-27/i686/./opcodes/.libs:/home/amylaar/bld/2006-11-27/i686/./prev-opcodes/.libs:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libstdc++-v3/.libs:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libmudflap/.libs:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libssp/.libs:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libgomp/.libs:/home/amylaar/bld/2006-11-27/i686/./gcc:/home/amylaar/bld/2006-11-27/i686/./prev-gcc

amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
Exception in thread "Thread-2" java.lang.NullPointerException
   at PR18699.update(PR18699.java:34)
   at java.util.Observable.notifyObservers(Observable.java:171)
   at PR18699.run(PR18699.java:26)
   at java.lang.Thread.run(Thread.java:682)
1
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
Exception in thread "Thread-1" ^[[Ajava.lang.NullPointerException
   at PR18699.update(PR18699.java:34)
   at java.util.Observable.notifyObservers(Observable.java:171)
   at PR18699.run(PR18699.java:26)
   at java.lang.Thread.run(Thread.java:682)
1
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
Exception in thread "Thread-2" ^[[A
java.lang.NullPointerException
   at PR18699.update(PR18699.java:34)
   at java.util.Observable.notifyObservers(Observable.java:171)
   at PR18699.run(PR18699.java:26)
   at java.lang.Thread.run(Thread.java:682)
1
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij
-classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0


-- 


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=30119