compare_test issue.

[Karel Gardas <kgardas at objectsecurity dot com>]

Hello,

Mike Stump from Apple recommend me to use compare_test utility from
contrib directory for comparison of test results of plain gcc3.3 and
gcc3.3 with his patch applied.

Unfortunately, IMHO I've found small bug in this utility since it reports
bad results for tests which are pretty identical.

I.e. I have two g++.sum files. diff -u bettween them looks:

--- gcc33/g++.sum	Wed Feb 19 14:38:12 2003
+++ gcc33-sp1/g++.sum	Wed Feb 19 10:51:09 2003
@@ -1,4 +1,4 @@
-Test Run By karel on Wed Feb 19 14:22:42 2003
+Test Run By karel on Wed Feb 19 10:35:33 2003
 Native configuration is i686-pc-linux-gnu

 		=== g++ tests ===
@@ -7,7 +7,7 @@
     unix

 Running target unix
-Running /home/karel/usr/src/cvs/gcc/gcc-3_3-branch/gcc/testsuite/g++.dg/bprob/bprob.exp ...
+Running /home/karel/usr/src/cvs/gcc/gcc33-test/gcc/testsuite/g++.dg/bprob/bprob.exp ...
 PASS: g++.dg/bprob/g++-bprob-1.C compilation,  -g  -fprofile-arcs
 PASS: g++.dg/bprob/g++-bprob-1.C execution,    -g  -fprofile-arcs
 PASS: g++.dg/bprob/g++-bprob-1.C compilation,  -g  -fbranch-probabilities
@@ -36,7 +36,7 @@
 PASS: g++.dg/bprob/g++-bprob-1.C execution,    -Os  -fprofile-arcs
 PASS: g++.dg/bprob/g++-bprob-1.C compilation,  -Os  -fbranch-probabilities
 PASS: g++.dg/bprob/g++-bprob-1.C execution,    -Os  -fbranch-probabilities
-Running /home/karel/usr/src/cvs/gcc/gcc-3_3-branch/gcc/testsuite/g++.dg/compat/compat.exp ...
+Running /home/karel/usr/src/cvs/gcc/gcc33-test/gcc/testsuite/g++.dg/compat/compat.exp ...
 PASS: g++.dg/compat/abi/bitfield1 main_tst.o compile
 PASS: g++.dg/compat/abi/bitfield1 x_tst.o compile
 PASS: g++.dg/compat/abi/bitfield1 y_tst.o compile
@@ -167,7 +167,7 @@
 PASS: g++.dg/compat/init/init-ref2 y_tst.o compile
 PASS: g++.dg/compat/init/init-ref2 x_tst.o-y_tst.o link
 PASS: g++.dg/compat/init/init-ref2 x_tst.o-y_tst.o execute
-Running /home/karel/usr/src/cvs/gcc/gcc-3_3-branch/gcc/testsuite/g++.dg/debug/debug.exp ...
+Running /home/karel/usr/src/cvs/gcc/gcc33-test/gcc/testsuite/g++.dg/debug/debug.exp ...
 PASS: g++.dg/debug/const1.C (test for excess errors)
 PASS: g++.dg/debug/const1.C (test for excess errors)
 PASS: g++.dg/debug/const1.C (test for excess errors)
@@ -492,7 +492,7 @@
 PASS: g++.dg/debug/typedef1.C (test for excess errors)
 PASS: g++.dg/debug/typedef1.C (test for excess errors)
 PASS: g++.dg/debug/typedef1.C (test for excess errors)
-Running /home/karel/usr/src/cvs/gcc/gcc-3_3-branch/gcc/testsuite/g++.dg/dg.exp ...
+Running /home/karel/usr/src/cvs/gcc/gcc33-test/gcc/testsuite/g++.dg/dg.exp ...
 PASS: g++.dg/abi/bitfield1.C (test for excess errors)
 PASS: g++.dg/abi/bitfield1.C execution test
 PASS: g++.dg/abi/bitfield2.C (test for excess errors)
@@ -1437,7 +1437,7 @@
 XFAIL: g++.dg/warn/return-reference.C  (test for bogus messages, line 10)
 PASS: g++.dg/warn/return-reference.C (test for excess errors)
 PASS: g++.dg/warn/weak1.C (test for excess errors)
-Running /home/karel/usr/src/cvs/gcc/gcc-3_3-branch/gcc/testsuite/g++.dg/gcov/gcov.exp ...
+Running /home/karel/usr/src/cvs/gcc/gcc33-test/gcc/testsuite/g++.dg/gcov/gcov.exp ...
 PASS: g++.dg/gcov/gcov-1.C (test for excess errors)
 PASS: g++.dg/gcov/gcov-1.C execution test
 PASS: g++.dg/gcov/gcov-1.C gcov
@@ -1447,7 +1447,7 @@
 PASS: g++.dg/gcov/gcov-3.C (test for excess errors)
 PASS: g++.dg/gcov/gcov-3.C execution test
 PASS: g++.dg/gcov/gcov-3.C gcov
-Running /home/karel/usr/src/cvs/gcc/gcc-3_3-branch/gcc/testsuite/g++.dg/special/ecos.exp ...
+Running /home/karel/usr/src/cvs/gcc/gcc33-test/gcc/testsuite/g++.dg/special/ecos.exp ...
 PASS: g++.dg/special/conpr-1.C (test for excess errors)
 PASS: g++.dg/special/conpr-1.C execution test
 PASS: g++.dg/special/conpr-2.C (test for excess errors)
@@ -1458,8 +1458,8 @@
 PASS: g++.dg/special/conpr-3.C execution test
 PASS: g++.dg/special/initp1.C (test for excess errors)
 PASS: g++.dg/special/initp1.C execution test
-Running /home/karel/usr/src/cvs/gcc/gcc-3_3-branch/gcc/testsuite/g++.dg/tls/tls.exp ...
-Running /home/karel/usr/src/cvs/gcc/gcc-3_3-branch/gcc/testsuite/g++.old-deja/old-deja.exp ...
+Running /home/karel/usr/src/cvs/gcc/gcc33-test/gcc/testsuite/g++.dg/tls/tls.exp ...
+Running /home/karel/usr/src/cvs/gcc/gcc33-test/gcc/testsuite/g++.old-deja/old-deja.exp ...
 PASS: g++.abi/aggregates.C (test for excess errors)
 PASS: g++.abi/aggregates.C  Execution test
 PASS: g++.abi/align.C (test for excess errors)
@@ -8103,6 +8103,6 @@
 # of expected failures		91
 # of untested testcases		9
 # of unsupported tests		3
-/home/karel/usr/src/cvs/gcc/obj/gcc/testsuite/../g++ version 3.3 20030218 (prerelease)
+/home/karel/usr/src/cvs/gcc/obj-33sp1/gcc/testsuite/../g++ version 3.3 20030218 (prerelease)



As you can see these tests differ only in path to directory where they
were executed.

The compare_test utility on the other hand produces (utility is from
yesterday version of gcc-3_3-branch):

thinkpad:~/cvs/gcc/test-results$ ~/cvs/gcc/gcc-3_3-branch/contrib/compare_tests gcc33/g++.sum gcc33-sp1/g++.sum
New tests that FAIL:

FAIL: g++.brendan/err-msg2.C conflicting linkage - (test for errors, line 6)
FAIL: g++.brendan/parse3.C - (test for bogus messages, line 22)
FAIL: g++.brendan/parse4.C - treated as decl (test for bogus messages, line 22)
FAIL: g++.brendan/parse4.C - treated as decl (test for bogus messages, line 23)
FAIL: g++.brendan/parse5.C - (test for bogus messages, line 24)
FAIL: g++.brendan/parse6.C - (test for bogus messages, line 12)
FAIL: g++.brendan/union1.C  Execution test
FAIL: g++.brendan/union1.C (test for excess errors)
FAIL: g++.bugs/900404_04.C , (test for errors, line 15)
FAIL: g++.dg/ext/lvalue1.C not an lvalue (test for errors, line 7)
FAIL: g++.dg/lookup/scoped1.C  (test for errors, line 20)
FAIL: g++.dg/other/anon-struct.C  (test for bogus messages, line 8)
FAIL: g++.dg/parse/angle-bracket.C  (test for bogus messages, line 7)
FAIL: g++.dg/template/qualttp20.C  (test for warnings, line 19)
FAIL: g++.dg/template/ttp3.C  (test for bogus messages, line 21)
FAIL: g++.dg/template/ttp3.C  (test for bogus messages, line 26)
FAIL: g++.dg/warn/Wunused-2.C  (test for warnings, line 5)
FAIL: g++.dg/warn/incomplete1.C  (test for warnings, line 19)
FAIL: g++.dg/warn/return-reference.C  (test for bogus messages, line 10)
FAIL: g++.jason/access8.C cannot convert to inh (test for errors, line 28)
FAIL: g++.jason/ambig3.C - late parsing (test for bogus messages, line 12)
FAIL: g++.jason/ambig3.C - late parsing (test for bogus messages, line 14)
FAIL: g++.jason/cond.C  (test for errors, line 20)
FAIL: g++.jason/cond.C  (test for errors, line 22)
FAIL: g++.jason/cond.C  (test for errors, line 25)
FAIL: g++.jason/cond.C  (test for errors, line 27)
FAIL: g++.jason/cond.C  (test for errors, line 30)
FAIL: g++.jason/cond.C  (test for errors, line 33)
FAIL: g++.jason/cond.C , (test for errors, line 11)
FAIL: g++.jason/cond.C , (test for errors, line 16)
FAIL: g++.jason/cond.C , (test for errors, line 9)
FAIL: g++.jason/rfg16.C  (test for bogus messages, line 4)
FAIL: g++.jason/rfg16.C  (test for bogus messages, line 5)
FAIL: g++.jason/template18.C  Execution test
FAIL: g++.jason/template18.C (test for excess errors)
FAIL: g++.law/array2.C (test for excess errors)
FAIL: g++.law/missed-error1.C mismatch (test for errors, line 19)
FAIL: g++.law/missed-error1.C mismatch (test for errors, line 20)
FAIL: g++.mike/p10769a.C  (test for bogus messages, line 34)
FAIL: g++.mike/p10769a.C  (test for bogus messages, line 43)
FAIL: g++.mike/p10769a.C  Execution test
FAIL: g++.mike/p7325.C  Execution test
FAIL: g++.niklas/t135.C (test for excess errors)
FAIL: g++.ns/template13.C not declared in bar - (test for errors, line 14)
FAIL: g++.ns/template13.C not declared in bar - (test for errors, line 19)
FAIL: g++.oliva/expr1.C - operator!= - (test for bogus messages, line 12)
FAIL: g++.oliva/expr2.C  Execution test
FAIL: g++.oliva/ext1.C  Execution test
FAIL: g++.oliva/linkage1.C (test for excess errors)
FAIL: g++.oliva/ns2.C - conflict - (test for bogus messages, line 13)
FAIL: g++.oliva/template10.C  (test for errors, line 22)
FAIL: g++.oliva/template9.C invalid initializer - (test for errors, line 10)
FAIL: g++.oliva/typeof1.C caused compiler crash
FAIL: g++.other/access2.C not accessible - (test for errors, line 7)
FAIL: g++.other/access2.C not accessible - (test for errors, line 9)
FAIL: g++.other/access6.C - private - (test for bogus messages, line 14)
FAIL: g++.other/access6.C - private - (test for bogus messages, line 9)
FAIL: g++.other/access6.C - redeclared - (test for bogus messages, line 11)
FAIL: g++.other/crash31.C (test for excess errors)
FAIL: g++.other/crash31.C parse error (test for errors, line 12)
FAIL: g++.other/crash32.C (test for excess errors)
FAIL: g++.other/decl1.C (test for excess errors)
FAIL: g++.other/decl5.C extra qualification (test for warnings, line 55)
FAIL: g++.other/decl5.C extra qualification (test for warnings, line 56)
FAIL: g++.other/enum5.C  Execution test
FAIL: g++.other/enum5.C (test for excess errors)
FAIL: g++.other/overload11.C (test for excess errors)
FAIL: g++.other/overload11.C not a call (test for warnings, line 85)
FAIL: g++.pt/const2.C  Execution test
FAIL: g++.pt/const2.C (test for excess errors)
FAIL: g++.pt/defarg8.C - before > - (test for bogus messages, line 13)
FAIL: g++.pt/explicit76.C - ICE - (test for bogus messages, line 11)
FAIL: g++.pt/friend44.C  Execution test
FAIL: g++.pt/friend44.C (test for excess errors)
FAIL: g++.pt/inherit2.C - (test for bogus messages, line 28)
FAIL: g++.pt/inherit2.C - (test for bogus messages, line 53)
FAIL: g++.pt/inherit2.C private - (test for errors, line 27)
FAIL: g++.pt/inherit2.C within this context - (test for errors, line 44)
FAIL: g++.pt/memtemp87.C caused compiler crash
FAIL: g++.pt/overload13.C  (test for bogus messages, line 14)
FAIL: g++.pt/overload13.C  Execution test
FAIL: g++.pt/overload7.C - instantiated from here - (test for bogus messages, line 19)
FAIL: g++.pt/overload7.C - no matching function for call - (test for bogus messages, line 14)
FAIL: g++.pt/overload8.C - must have argument of class type - (test for bogus messages, line 10)
FAIL: g++.pt/ptrmem5.C - cannot convert - (test for bogus messages, line 17)
FAIL: g++.pt/spec24.C missing template header - (test for errors, line 5)
FAIL: g++.pt/typename12.C  Execution test
FAIL: g++.pt/typename12.C (test for excess errors)
FAIL: g++.pt/typename13.C (test for excess errors)
FAIL: g++.robertl/eb125.C (test for excess errors)
FAIL: g++.robertl/eb17.C  Execution test
FAIL: g++.robertl/eb17.C (test for excess errors)

Old tests that failed, that have disappeared: (Eeek!)

FAIL: g++.brendan/err-msg2.C conflicting linkage - (test for errors, line 6)
FAIL: g++.brendan/parse3.C - (test for bogus messages, line 22)
FAIL: g++.brendan/parse4.C - treated as decl (test for bogus messages, line 22)
FAIL: g++.brendan/parse4.C - treated as decl (test for bogus messages, line 23)
FAIL: g++.brendan/parse5.C - (test for bogus messages, line 24)
FAIL: g++.brendan/parse6.C - (test for bogus messages, line 12)
FAIL: g++.brendan/union1.C  Execution test
FAIL: g++.brendan/union1.C (test for excess errors)
FAIL: g++.bugs/900404_04.C , (test for errors, line 15)
FAIL: g++.dg/ext/lvalue1.C not an lvalue (test for errors, line 7)
FAIL: g++.dg/lookup/scoped1.C  (test for errors, line 20)
FAIL: g++.dg/other/anon-struct.C  (test for bogus messages, line 8)
FAIL: g++.dg/parse/angle-bracket.C  (test for bogus messages, line 7)
FAIL: g++.dg/template/qualttp20.C  (test for warnings, line 19)
FAIL: g++.dg/template/ttp3.C  (test for bogus messages, line 21)
FAIL: g++.dg/template/ttp3.C  (test for bogus messages, line 26)
FAIL: g++.dg/warn/Wunused-2.C  (test for warnings, line 5)
FAIL: g++.dg/warn/incomplete1.C  (test for warnings, line 19)
FAIL: g++.dg/warn/return-reference.C  (test for bogus messages, line 10)
FAIL: g++.jason/access8.C cannot convert to inh (test for errors, line 28)
FAIL: g++.jason/ambig3.C - late parsing (test for bogus messages, line 12)
FAIL: g++.jason/ambig3.C - late parsing (test for bogus messages, line 14)
FAIL: g++.jason/cond.C  (test for errors, line 20)
FAIL: g++.jason/cond.C  (test for errors, line 22)
FAIL: g++.jason/cond.C  (test for errors, line 25)
FAIL: g++.jason/cond.C  (test for errors, line 27)
FAIL: g++.jason/cond.C  (test for errors, line 30)
FAIL: g++.jason/cond.C  (test for errors, line 33)
FAIL: g++.jason/cond.C , (test for errors, line 11)
FAIL: g++.jason/cond.C , (test for errors, line 16)
FAIL: g++.jason/cond.C , (test for errors, line 9)
FAIL: g++.jason/rfg16.C  (test for bogus messages, line 4)
FAIL: g++.jason/rfg16.C  (test for bogus messages, line 5)
FAIL: g++.jason/template18.C  Execution test
FAIL: g++.jason/template18.C (test for excess errors)
FAIL: g++.law/array2.C (test for excess errors)
FAIL: g++.law/missed-error1.C mismatch (test for errors, line 19)
FAIL: g++.law/missed-error1.C mismatch (test for errors, line 20)
FAIL: g++.mike/p10769a.C  (test for bogus messages, line 34)
FAIL: g++.mike/p10769a.C  (test for bogus messages, line 43)
FAIL: g++.mike/p10769a.C  Execution test
FAIL: g++.mike/p7325.C  Execution test
FAIL: g++.niklas/t135.C (test for excess errors)
FAIL: g++.ns/template13.C not declared in bar - (test for errors, line 14)
FAIL: g++.ns/template13.C not declared in bar - (test for errors, line 19)
FAIL: g++.oliva/expr1.C - operator!= - (test for bogus messages, line 12)
FAIL: g++.oliva/expr2.C  Execution test
FAIL: g++.oliva/ext1.C  Execution test
FAIL: g++.oliva/linkage1.C (test for excess errors)
FAIL: g++.oliva/ns2.C - conflict - (test for bogus messages, line 13)
FAIL: g++.oliva/template10.C  (test for errors, line 22)
FAIL: g++.oliva/template9.C invalid initializer - (test for errors, line 10)
FAIL: g++.oliva/typeof1.C caused compiler crash
FAIL: g++.other/access2.C not accessible - (test for errors, line 7)
FAIL: g++.other/access2.C not accessible - (test for errors, line 9)
FAIL: g++.other/access6.C - private - (test for bogus messages, line 14)
FAIL: g++.other/access6.C - private - (test for bogus messages, line 9)
FAIL: g++.other/access6.C - redeclared - (test for bogus messages, line 11)
FAIL: g++.other/crash31.C (test for excess errors)
FAIL: g++.other/crash31.C parse error (test for errors, line 12)
FAIL: g++.other/crash32.C (test for excess errors)
FAIL: g++.other/decl1.C (test for excess errors)
FAIL: g++.other/decl5.C extra qualification (test for warnings, line 55)
FAIL: g++.other/decl5.C extra qualification (test for warnings, line 56)
FAIL: g++.other/enum5.C  Execution test
FAIL: g++.other/enum5.C (test for excess errors)
FAIL: g++.other/overload11.C (test for excess errors)
FAIL: g++.other/overload11.C not a call (test for warnings, line 85)
FAIL: g++.pt/const2.C  Execution test
FAIL: g++.pt/const2.C (test for excess errors)
FAIL: g++.pt/defarg8.C - before > - (test for bogus messages, line 13)
FAIL: g++.pt/explicit76.C - ICE - (test for bogus messages, line 11)
FAIL: g++.pt/friend44.C  Execution test
FAIL: g++.pt/friend44.C (test for excess errors)
FAIL: g++.pt/inherit2.C - (test for bogus messages, line 28)
FAIL: g++.pt/inherit2.C - (test for bogus messages, line 53)
FAIL: g++.pt/inherit2.C private - (test for errors, line 27)
FAIL: g++.pt/inherit2.C within this context - (test for errors, line 44)
FAIL: g++.pt/memtemp87.C caused compiler crash
FAIL: g++.pt/overload13.C  (test for bogus messages, line 14)
FAIL: g++.pt/overload13.C  Execution test
FAIL: g++.pt/overload7.C - instantiated from here - (test for bogus messages, line 19)
FAIL: g++.pt/overload7.C - no matching function for call - (test for bogus messages, line 14)
FAIL: g++.pt/overload8.C - must have argument of class type - (test for bogus messages, line 10)
FAIL: g++.pt/ptrmem5.C - cannot convert - (test for bogus messages, line 17)
FAIL: g++.pt/spec24.C missing template header - (test for errors, line 5)
FAIL: g++.pt/typename12.C  Execution test
FAIL: g++.pt/typename12.C (test for excess errors)
FAIL: g++.pt/typename13.C (test for excess errors)
FAIL: g++.robertl/eb125.C (test for excess errors)
FAIL: g++.robertl/eb17.C  Execution test
FAIL: g++.robertl/eb17.C (test for excess errors)


Have I done something wrong?

Thanks,

Karel
--
Karel Gardas                  kgardas at objectsecurity dot com
ObjectSecurity Ltd.           http://www.objectsecurity.com