Analysis of Mauve failures - Part 2

Wed Apr 3 08:33:00 GMT 2002

Hi,

Thanks for all the feedback on Part 1.
Since it it so popular here is Part 2!
Note again that the analysis is not very deep,
it is mainly a tool for me to get a list of issues to look into.

> FAIL: gnu.testlet.java.net.DatagramSocket.DatagramSocketTest: Error : test_echoWithTimeout failed - 5 Exception occured while sending/receiving  (number 1)
We get a NullPointerException after the DatagramSocket has been closed
and receive() is called. Don't know why since impl is not null.

> FAIL: gnu.testlet.java.net.InetAddress.InetAddressTest: Error : test_Basics failed - 10  Should not throw UnknownHostException here  (number 1)
> FAIL: gnu.testlet.java.net.InetAddress.InetAddressTest: test_Basics failed - 11 - NullPointerException (number 1)
> FAIL: gnu.testlet.java.net.InetAddress.InetAddressTest: test_Basics failed - 12 - NullPointerException (number 1)
Tests for host hpjavaux.cup.hp.com which does not exist. When changed to
lookup savannah.gnu.org all tests succeed.

> FAIL: gnu.testlet.java.net.ServerSocket.ServerSocketTest: Error : test_params failed - 4toString did not return proper values  (number 1)
ServerSocket.toString() returns:
  ServerSocket [addr=0.0.0.0/0.0.0.0,port=0,localport=30000]
the test expects:
  ServerSocket[addr=all-zeros-broadcast/0.0.0.0,port=0,localport=30000]
Since the API does not specify any format I think the test is wrong.

> FAIL: gnu.testlet.java.net.Socket.SocketTest: Error : test_Basics failed - 7 exception should not have been thrown here (number 1)
> FAIL: gnu.testlet.java.net.Socket.SocketTest: Error : test_params failed - 10 exception was thrown (number 1)
> FAIL: gnu.testlet.java.net.Socket.SocketTest: Error : test_BasicServer failed - 7 len returned is not correct.  1 (number 1)
> WARNING: program timed out.
This test does not run at all when run by hand. Have to figure out how
the setup is different. But since www.cygnus.com != 205.180.83.41
anymore some failures are expected.

> FAIL: gnu.testlet.java.net.URL.URLTest: openConnection (number 1)
> WARNING: program timed out.
Seems to test against
http://sourceware.cygnus.com:80/mauve/testarea/index.html
Which is now out of date since the failures read:
got 'Apache/1.3.23 (Unix)' but expected 'Apache/1.3.9 (Unix)'
got 'text/html; charset=iso-8859-1' but expected 'text/html'
got '<!DOCT' but expected '<HTML>'

> FAIL: Link for gnu.testlet.java.text.AttributedCharacterIterator.implement
Works fine when done by hand. Probably some setup lameness.

> FAIL: gnu.testlet.java.text.AttributedString.Test: Attribute key count (number 1)
Checks that getAllAttributeKeys() returns 1 when it has just added 2
Attributes. Is this test correct?

> FAIL: gnu.testlet.java.text.DateFormatSymbols.Test: invalid locale (number 1)
Our ResourceBundle.getBundle() returns something for Locale("yi", "yi").
Is that actually wrong?

> FAIL: gnu.testlet.java.text.SimpleDateFormat.regress: CDT (number 1)
> FAIL: gnu.testlet.java.text.SimpleDateFormat.regress: EDT (number 1)
> FAIL: gnu.testlet.java.text.SimpleDateFormat.regress: EST (number 1)
> FAIL: gnu.testlet.java.text.SimpleDateFormat.regress: PDT (number 1)
got 'Fri, 18 May 2001 12:18:06 CDT' but expected
    'Fri, 18 May 2001 17:18:06 GMT'
got 'Fri, 18 May 2001 13:18:06 EDT' but expected
    'Fri, 18 May 2001 17:18:06 GMT'
got 'Fri, 18 May 2001 13:18:06 EDT' but expected
    'Fri, 18 May 2001 17:18:06 GMT'
got 'Fri, 18 May 2001 10:18:06 PDT' but expected
    'Fri, 18 May 2001 17:18:06 GMT'
Are we missing a Calendar.setTimeZone() somewere?

> FAIL: Link for gnu.testlet.java.util.AbstractCollection.AcuniaAbstractCollectionTest
> FAIL: Link for gnu.testlet.java.util.AbstractList.AcuniaAbstractListTest
> FAIL: Link for gnu.testlet.java.util.AbstractMap.AcuniaAbstractMapTest
> FAIL: Link for gnu.testlet.java.util.AbstractMap.AcuniaExAbstractMap
> FAIL: Link for gnu.testlet.java.util.AbstractSequentialList.AcuniaAbstractSequentialListTest
> FAIL: Link for gnu.testlet.java.util.AbstractSet.AcuniaAbstractSetTest
The setup has problems with tests that use more than one class for the
Testlet. Will rewrite testlets to be self contained.

> FAIL: gnu.testlet.java.util.ArrayList.AcuniaArrayListTest: should throw an IndexOutOfBoundsException -- 3 (number 1)
removeRange(3,2) called on an ArrayList of 14 elements.
Seems reasonable but spec does not explicitly specify this behaviour.

> FAIL: gnu.testlet.java.util.ArrayList.AcuniaArrayListTest: ArrayList must not be changed -- 3 (number 1)
Probably fallout from last failure.

> FAIL: gnu.testlet.java.util.ArrayList.AcuniaArrayListTest: should throw an IndexOutOfBoundsException -- 5 (number 1)
> FAIL: gnu.testlet.java.util.ArrayList.AcuniaArrayListTest: ArrayList must not be changed -- 5 (number 1)
Same as above removeRange(15,13).

> FAIL: gnu.testlet.java.util.ArrayList.AcuniaArrayListTest: should throw a ConcurrentModificationException -- 7 (number 1)
set(int, Object) does not make the Iterator throw a
ConcurrentModificationException. Seems to not be required by the spec
though. Do people depend on this behaviour?

> FAIL: gnu.testlet.java.util.Arrays.sort (number 1)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 2)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 3)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 4)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 5)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 6)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 7)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 8)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 9)
> FAIL: gnu.testlet.java.util.Arrays.sort (number 10)
Ugh. Our sort does not sort...
Some example failures:

- 79,84,94,42,84,75,11,52,84,18,69,76,14,69,32,28,49,87,98,39,.
+ 11,14,18,28,32,39,42,49,52,69,79,69,75,76,84,84,84,87,94,98

- 80,19,8,10,44,84,61,56,56,6,16,57,59,28,12,92,55,34,1,80,.
+ 1,6,8,10,12,16,19,28,34,44,55,56,56,57,59,61,80,80,92,84

- 68,82,69,40,68,44,11,89,87,92,62,71,52,86,51,23,92,42,13,65,.
+ 11,13,23,40,42,44,51,52,62,65,68,71,68,69,82,86,87,89,92,92

> FAIL: gnu.testlet.java.util.HashMap.AcuniaHashMapTest: checking value of loadFactor, got 1.5 (number 1)
Strange it creates a new HashMap(10,1.5f) and then expects to get a
loadFactor of 0.75f !?!

> FAIL: gnu.testlet.java.util.HashMap.AcuniaHashMapTest: it.hasNext should not throw ConcurrentModificationException (number 1)
Again a bogus test? It gets an iterator, removes an element and the
expects it.hasNext() not to throw and ConcurrentModificationException
!?!

> FAIL: gnu.testlet.java.util.HashMap.AcuniaHashMapTest: should throw an UnsupportedOperationException (number 1)
Seems fallout from the last bogus test.

> FAIL: gnu.testlet.java.util.Hashtable.AcuniaHashtableTest: should throw NullPointerException (number 1)
Our docs say: "This is the new API for the old contains(), except that
it is forgiving of null."
But the spec says "Note that this method is identical in functionality
to contains"

> FAIL: gnu.testlet.java.util.Hashtable.AcuniaHashtableTest: should throw NullPointerException -- 1 (number 1)
remove(null). Our docs say "Map.remove and Dictionary.remove disagree
whether null is a valid parameter; at the moment, this implementation
obeys Map.remove, and silently ignores null."

> FAIL: gnu.testlet.java.util.LinkedList.AcuniaLinkedListTest: should throw a ConcurrentModificationException -- 7 (number 1)
Same issue as with ArrayList.
set(int, Object) does not make the Iterator throw a
ConcurrentModificationException. Seems to not be required by the spec
though. Do people depend on this behaviour?

That was it for now.
Stay tuned for Part 3 in the never ending Mauve failures soap.

Cheers,

Mark