Bug 30119 - libjava testsuite output is erratic and unhelpful
Summary: libjava testsuite output is erratic and unhelpful
Status: RESOLVED WONTFIX
Alias: None
Product: gcc
Classification: Unclassified
Component: testsuite (show other bugs)
Version: 4.3.0
: P3 normal
Target Milestone: ---
Assignee: Not yet assigned to anyone
URL:
Keywords:
Depends on:
Blocks: 29842 29858
  Show dependency tree / graph
 
Reported: 2006-12-08 11:39 UTC by Jorn Wolfgang Rennecke
Modified: 2016-12-15 21:40 UTC (History)
4 users (show)

See Also:
Host:
Target:
Build:
Known to work:
Known to fail:
Last reconfirmed:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Jorn Wolfgang Rennecke 2006-12-08 11:39:53 UTC 
I have tested the patch for PR rtl-optimization/29858 in revision 119261
on gcc01 (i686-pc-linux-gnu)
Compared to a pristine build of revision 119055, these are the
additional failures:

> FAIL: gcc.dg/visibility-11.c scan-assembler memcpy@PLT
14a16,17
> FAIL: gcc.dg/vect/vect-pow-1.c scan-tree-dump pattern recognized
> FAIL: gcc.dg/vect/vect-pow-2.c scan-tree-dump pattern recognized
107a111,112
> FAIL: PR18699 execution - gij test
> FAIL: PR18699 execution - gij test

compared to a pristine build of 119261 (the base version), this are the
aditional failures:

110a111,115
> FAIL: PR18699 execution - gij test
> FAIL: PR18699 execution - gij test
> FAIL: SyncTest execution - gij test
> FAIL: SyncTest execution - gij test
> FAIL: SyncTest execution - gij test

With failures in the gcc core or c++ / libstdc++ testsuite, reproducing
failures is very straight forward : you cut & paste the appropriate line(s)
from the log file in order, and can thus ovserve the failure interactively,
and then use gdb and/or debugging dumps to further investigate.

The log file shows this about the PR18699 failure:
PASS: PR18699 -O3 output - source compiled test
byte compile: /home/amylaar/bld/2006-11-27-29858/i686/gcc/gcj -B/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/ -B/home/amylaar/bld/2006-11-27-29858/i686/gcc/ --encoding=UTF-8 -C -I/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite/../libgcj-4.3.0.jar -g /home/amylaar/bld/2006-11-27-29858/srcw/libjava/testsuite/libjava.lang/PR18699.java -d /home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite 2>@ stdout
PASS: PR18699 byte compilation
PR18699PR18699 set_ld_library_path_env_vars: ld_library_path=.:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27-29858/i686/gcc
Setting LD_LIBRARY_PATH to .:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27-29858/i686/gcc:.:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27-29858/i686/gcc:.:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27-29858/i686/gcc:/home/amylaar/bld/2006-11-27-29858/i686/./bfd/.libs:/home/amylaar/bld/2006-11-27-29858/i686/./prev-bfd/.libs:/home/amylaar/bld/2006-11-27-29858/i686/./opcodes/.libs:/home/amylaar/bld/2006-11-27-29858/i686/./prev-opcodes/.libs:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libstdc++-v3/.libs:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libmudflap/.libs:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libssp/.libs:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libgomp/.libs:/home/amylaar/bld/2006-11-27-29858/i686/./gcc:/home/amylaar/bld/2006-11-27-29858/i686/./prev-gcc
FAIL: PR18699 execution - gij test

When I cut & past the line above 'PASS: PR18699 byte compilation', a file named
'@' is created, which contains:

gcj: stdout: No such file or directory

The only README file in the entire libjava testsuite is testsuite/libjava.verify/README.verify .

The web documentation on testing http://gcc.gnu.org/install/test.html only
has the basic meaning of PASS/ FAIL etc for the benefit of a person who
installs the library without modifying any pieces of the GNU compiler
collection.

I should not be required to reverse-engineer the libjava testsuite before
I can interpret/debug test results for my patches to the gcc core.
There should be easy-to-follow documentation how I can get from
the debugging log to reproducing the failure.

Moreover, considering that there is no regession against the baseline for any other part of the GNU compiler collection, and the recent track record of
libjava testing, it seems highly likely that the regressions are actually testsuite failures.
Comment 1 Andrew Haley 2006-12-08 11:46:13 UTC 
It's not necessary to do any I/O redirection when byte compiling: just execute the command itself without the "2>" etc.
Comment 2 Jorn Wolfgang Rennecke 2006-12-08 14:59:16 UTC 
(In reply to comment #1)
> It's not necessary to do any I/O redirection when byte compiling: just execute
> the command itself without the "2>" etc.
> 

All right, that gives me a file 'PR18699.class'  But how is this actually executed?
The next two lines of the log file seem to be only about setting environment variables, and the very next line has the 'FAIL' message.
Comment 3 Andrew Haley 2006-12-08 15:20:49 UTC 
Run it with the libjava/gij PR18699.class
Comment 4 Jorn Wolfgang Rennecke 2006-12-08 15:46:18 UTC 
(In reply to comment #3)
> Run it with the libjava/gij PR18699.class

ls -l PR18699.class
-rw-r--r--  1 amylaar users 1688 2006-12-08 15:52 PR18699.class
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij PR18699.class
Exception in thread "main" java.lang.NoClassDefFoundError: PR18699.class
   at gnu.java.lang.MainThread.run(MainThread.java:102)
Caused by: java.lang.ClassNotFoundException: PR18699.class not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:./], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
   at java.net.URLClassLoader.findClass(URLClassLoader.java:1080)
   at gnu.gcj.runtime.SystemClassLoader.findClass(natSystemClassLoader.cc:27)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:294)
   at gnu.java.lang.MainThread.run(MainThread.java:98)

gcc01 does not have jcf-dump, but when I scp the file to my local machine,
I see after the constant table:

Access flags: 0x20 super
This class: 2=PR18699, super: 4=java.util.Observable
Interfaces (count: 2):
- Implements: 6=java.lang.Runnable
- Implements: 8=java.util.Observer

So is this a bug in gij not to find the class in PR18699.class?
Comment 5 Andrew Haley 2006-12-08 16:45:12 UTC 
Set the classpath

gij -classpath <path to  PR18699.class>:<path to libgcj.jar>  PR18699
Comment 6 Jorn Wolfgang Rennecke 2006-12-13 13:25:21 UTC 
(In reply to comment #5)
> Set the classpath
> 
> gij -classpath <path to  PR18699.class>:<path to libgcj.jar>  PR18699

Unfortunately, that still doesn't work:
 
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath PR18699.class:../libgcj-4.3.0.jar PR18699.class 
Exception in thread "main" java.lang.NoClassDefFoundError: PR18699.class
   at gnu.java.lang.MainThread.run(MainThread.java:102)
Caused by: java.lang.ClassNotFoundException: PR18699.class not found in gnu.gcj.runtime.SystemClassLoader{urls=[file:PR18699.class,file:../libgcj-4.3.0.jar], parent=gnu.gcj.runtime.ExtensionClassLoader{urls=[], parent=null}}
   at java.net.URLClassLoader.findClass(URLClassLoader.java:1080)
   at gnu.gcj.runtime.SystemClassLoader.findClass(natSystemClassLoader.cc:27)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:294)
   at gnu.java.lang.MainThread.run(MainThread.java:98)
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ ls -l PR18699.class
-rw-r--r--  1 amylaar users 1688 2006-12-13 14:11 PR18699.class
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ ls -l ../libgcj-4.3.0.jar
-rw-r--r--  1 amylaar users 8547070 2006-12-05 00:07 ../libgcj-4.3.0.jar
Comment 7 Andrew Haley 2006-12-13 13:42:53 UTC 
You wrote

../gij -classpath PR18699.class:../libgcj-4.3.0.jar PR18699.class

But I wrote:

../gij -classpath .:../libgcj-4.3.0.jar PR18699
Comment 8 Jorn Wolfgang Rennecke 2006-12-13 19:47:15 UTC 
(In reply to comment #7)

Sorry, I misunderstood you.  Now the test works - most of the time:
 
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ /home/amylaar/bld/2006-11-27-29858/i686/gcc/gcj -B/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/ -B/home/amylaar/bld/2006-11-27-29858/i686/gcc/ --encoding=UTF-8 -C -I/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite/../libgcj-4.3.0.jar -g /home/amylaar/bld/2006-11-27-29858/srcw/libjava/testsuite/libjava.lang/PR18699.java -d /home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ export ld_library_path=.:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27-29858/i686/gcc
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ export LD_LIBRARY_PATH=.:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27-29858/i686/gcc:.:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27-29858/i686/gcc:.:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27-29858/i686/gcc:/home/amylaar/bld/2006-11-27-29858/i686/./bfd/.libs:/home/amylaar/bld/2006-11-27-29858/i686/./prev-bfd/.libs:/home/amylaar/bld/2006-11-27-29858/i686/./opcodes/.libs:/home/amylaar/bld/2006-11-27-29858/i686/./prev-opcodes/.libs:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libstdc++-v3/.libs:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libmudflap/.libs:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libssp/.libs:/home/amylaar/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libgomp/.libs:/home/amylaar/bld/2006-11-27-29858/i686/./gcc:/home/amylaar/bld/2006-11-27-29858/i686/./prev-gcc
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699
Exception in thread "Thread-2" java.lang.NullPointerException
   at PR18699.update(PR18699.java:34)
   at java.util.Observable.notifyObservers(Observable.java:171)
   at PR18699.run(PR18699.java:26)
   at java.lang.Thread.run(Thread.java:682)
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ echo $?
0
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ echo $?
0
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ echo $?
0
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ echo $?
0

I have no idea how to make it repeat the null pointer exception, except trying very often.  Here is another one:

amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699
Exception in thread "Thread-1" java.lang.NullPointerException
   at PR18699.update(PR18699.java:34)
   at java.util.Observable.notifyObservers(Observable.java:171)
   at PR18699.run(PR18699.java:26)
   at java.lang.Thread.run(Thread.java:682)
amylaar@gcc01:~/bld/2006-11-27-29858/i686/i686-pc-linux-gnu/libjava/testsuite$ free
             total       used       free     shared    buffers     cached
Mem:       1036088     969192      66896          0     448352     147260
-/+ buffers/cache:     373580     662508
Swap:       755012          0     755012
Comment 9 Andrew Haley 2006-12-15 11:21:33 UTC 
OK, so it looks like this is not a testsuite problem: the problem really is with the program being tested.

The next thing to do is run the program with GDB and find out where the SEGV occurs.
Comment 10 Jorn Wolfgang Rennecke 2006-12-19 12:57:18 UTC 
(In reply to comment #9)
> OK, so it looks like this is not a testsuite problem: the problem really is
> with the program being tested.

It is also a testsuite problem when the tests are randonly passing or failing;
such tests are not suitable as such in an automated testsuite.
I suppose you could write a wrapper to run them several times to get a
reasonable high confidence that the test generally passes - if the test does
not take too long - and maybe the probabilities can also be enhanced by
forking specifically designed other programs to perturb the load, but
unless/ until such infrastructure is put in place, these randomly failing
tests make it effectively impossible to pass a regression test.

I've redone to tests on my baseline where the test passed during the automated
test, and the results are again random:

amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ /home/amylaar/bld/2006-11-27/i686/gcc/gcj -B/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/ -B/home/amylaar/bld/2006-11-27/i686/gcc/ --encoding=UTF-8 -C -I/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite/../libgcj-4.3.0.jar -g /home/amylaar/bld/2006-11-27/srcw/libjava/testsuite/libjava.lang/PR18699.java -d /home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ export ld_library_path=.:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27/i686/gcc
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ export LD_LIBRARY_PATH=.:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27/i686/gcc:.:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27/i686/gcc:.:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/./libjava/.libs:/home/amylaar/bld/2006-11-27/i686/gcc:/home/amylaar/bld/2006-11-27/i686/./bfd/.libs:/home/amylaar/bld/2006-11-27/i686/./prev-bfd/.libs:/home/amylaar/bld/2006-11-27/i686/./opcodes/.libs:/home/amylaar/bld/2006-11-27/i686/./prev-opcodes/.libs:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libstdc++-v3/.libs:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libmudflap/.libs:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libssp/.libs:/home/amylaar/bld/2006-11-27/i686/i686-pc-linux-gnu/libgomp/.libs:/home/amylaar/bld/2006-11-27/i686/./gcc:/home/amylaar/bld/2006-11-27/i686/./prev-gcc

amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
Exception in thread "Thread-2" java.lang.NullPointerException
   at PR18699.update(PR18699.java:34)
   at java.util.Observable.notifyObservers(Observable.java:171)
   at PR18699.run(PR18699.java:26)
   at java.lang.Thread.run(Thread.java:682)
1
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
Exception in thread "Thread-1" ^[[Ajava.lang.NullPointerException
   at PR18699.update(PR18699.java:34)
   at java.util.Observable.notifyObservers(Observable.java:171)
   at PR18699.run(PR18699.java:26)
   at java.lang.Thread.run(Thread.java:682)
1
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
Exception in thread "Thread-2" ^[[A
java.lang.NullPointerException
   at PR18699.update(PR18699.java:34)
   at java.util.Observable.notifyObservers(Observable.java:171)
   at PR18699.run(PR18699.java:26)
   at java.lang.Thread.run(Thread.java:682)
1
amylaar@gcc01:~/bld/2006-11-27/i686/i686-pc-linux-gnu/libjava/testsuite$ ../gij -classpath .:../libgcj-4.3.0.jar PR18699;echo $?
0
Comment 11 Jeffrey A. Law 2016-12-15 21:40:45 UTC 
libjava is dead.