Results for g++ 3.3 application testing on i686-pc-linux-gnu

[poschmid at lbl dot gov]

I ran many tests on my i686-pc-linux-gnu computer, the CPU is
a Pentium IV with a 1.8 GHz clock.

Most of the problems I found, were optimizer related. It takes enormous
amounts of memory to compile blitz's expression templates, when the
optimization is turned on. The performance of the stepanov tests is
not coherent at optimization levels below -O3, and the results of
the FTensor tests are disappointing for some tests. The c++
implementation of the complex test in oopack is substantially, 13
times, slower than the implementation in c. An optimized build 
at the -O3 level miscompiles STLport's eh test suite resulting in a
crash when run. 

Three of the twelve boost regression test suite failures are
regressions with respect to gcc 3.2. Perhaps some of them could be
fixed before gcc 3.3 is released. (confer PR)
  
Here are the results in detail:

stepanov_v1p2.C: 

-O

test      absolute   additions      ratio with
number    time       per second     test0

 0        0.15sec    333.33M         1.00
 1        0.15sec    333.33M         1.00
 2        0.14sec    357.14M         0.93
 3        0.15sec    333.33M         1.00
 4        0.17sec    294.12M         1.13
 5        0.16sec    312.50M         1.07
 6        0.18sec    277.78M         1.20
 7        0.16sec    312.50M         1.07
 8        0.17sec    294.12M         1.13
 9        0.16sec    312.50M         1.07
10        0.16sec    312.50M         1.07
11        0.15sec    333.33M         1.00
12        0.17sec    294.12M         1.13
mean:     0.16sec    314.72M         1.06

Total absolute time: 2.07 sec

Abstraction Penalty: 1.06


-O2
test      absolute   additions      ratio with
number    time       per second     test0

 0        0.14sec    357.14M         1.00
 1        0.14sec    357.14M         1.00
 2        0.17sec    294.12M         1.21
 3        0.14sec    357.14M         1.00
 4        0.15sec    333.33M         1.07
 5        0.15sec    333.33M         1.07
 6        0.14sec    357.14M         1.00
 7        0.14sec    357.14M         1.00
 8        0.15sec    333.33M         1.07
 9        0.15sec    333.33M         1.07
10        0.15sec    333.33M         1.07
11        0.14sec    357.14M         1.00
12        0.15sec    333.33M         1.07
mean:     0.15sec    340.82M         1.05

Total absolute time: 1.91 sec

Abstraction Penalty: 1.05

-O3

test      absolute   additions      ratio with
number    time       per second     test0

 0        0.15sec    333.33M         1.00
 1        0.14sec    357.14M         0.93
 2        0.15sec    333.33M         1.00
 3        0.14sec    357.14M         0.93
 4        0.14sec    357.14M         0.93
 5        0.15sec    333.33M         1.00
 6        0.14sec    357.14M         0.93
 7        0.15sec    333.33M         1.00
 8        0.14sec    357.14M         0.93
 9        0.14sec    357.14M         0.93
10        0.14sec    357.14M         0.93
11        0.15sec    333.33M         1.00
12        0.14sec    357.14M         0.93
mean:     0.14sec    347.79M         0.96

Total absolute time: 1.87 sec

Abstraction Penalty: 0.96


OOPACK Version 1.7:
g++ -O3
Max=100000 Matrix=1000 Complex=100000 Iterator=100000

Test       Iterations     C    OOP     C    OOP  Ratio
----       ----------  -----------  -----------  -----
Max            100000    1.2   1.6   84.0  64.1    1.3
Matrix           1000    0.4   0.4  641.0 609.8    1.1
Complex        100000    1.1  14.8  727.3  54.2   13.4
Iterator       100000    0.3   0.3  689.7 645.2    1.1

Current boost cvs: 
CXXFLAGS = -c -Wall -fabi-version=0 -ftemplate-depth-100   -g -O0 -fno-inline

Twelve tests fail. 
libs/date_time/test/testtime_resolution_traits.cpp * +		
libs/function/test/sum_avg_portable.cpp 
libs/math/octonion/octonion_test.cpp 
libs/math/quaternion/quaternion_test.cpp 
libs/random/random_test.cpp *
libs/static_assert/static_assert_example_1.cpp +
libs/test/test/errors_handling_test.cpp
libs/test/test/test_fp_comparisons.cpp
libs/type_traits/test/is_class_test.cpp
libs/type_traits/test/is_empty_test.cpp
libs/type_traits/test/is_union_test.cpp	*
libs/type_traits/test/tricky_function_type_test.cpp *

The ones marked with a star are regressions with respect to gcc
3.2. The rejections of testtime_resolution_traits.cpp and
static_assert_example_1.cpp are because of coding errors.

blitz-0.6: Compiling the files located in the examples directory
requires enormous amounts of virtual memory. All of the available
virtual memory, 768 MB, on my machine is depleted when trying to
compile an optimized build of the source files array and matmult at
the O2 level. All other examples compile and run fine.

root_v3.05.01:
Everything works fine.

pooma-gcc:
No problems detected except that the perl script aborts shortly before
writing the summary table.

Code from Josuttis' Book "The C++ Standard Library": 
Everything works fine, except for the problem described in PR
libstcd++/6745 and two known parser problems exposed by compiling
sortset and sortvec.

Code from Vandevoordes' and Josuttis' Book "C++ Templates": 
According to the authors the source codes stack7test and max3b are
illegal and should therefore be rejected by the compiler. gcc 3.3 and
3.4 accept them, though. 

Apart from that, I detected no problems. 

mtl-2.1.2-21:
No problems detected, after fixing some coding glitches. The regression test
suite passes without a problem.

Ace and Tao version 5.2.1.
Ace and Tao compile without a problem. I ran many of the included
tests, except for a core file generated by running
Logging_Strategy_Test, I detected no glitches.

STLport-5.0-0409
The libraries compile fine but the eh test suite crashes in "[deque]
n-size constructor (const)". This happens only when the -O3
optimization is active while compiling the test suite.  

FTensor--main--1.1pre20: No problems detected. All tests pass. On some
speed tests the performance of the optimizer is quite poor. Whereas
for the seventh, eighth and ninth tests the tensor based
implementation runs faster.

CXXFLAGS= -ftemplate-depth-1000 -Drestrict=__restrict__ -O3
-finline-functions -finline-limit-1000 -funroll-loops 

fast 1

real	0m0.760s
user	0m0.760s
sys	0m0.000s
Tensor 1

real	0m0.764s
user	0m0.770s
sys	0m0.000s
fast 2

real	0m1.812s
user	0m1.790s
sys	0m0.000s
Tensor 2

real	0m4.832s
user	0m4.800s
sys	0m0.000s
fast 3

real	0m0.422s
user	0m0.420s
sys	0m0.000s
Tensor 3

real	0m1.743s
user	0m1.750s
sys	0m0.000s
fast 4

real	0m0.730s
user	0m0.730s
sys	0m0.000s
Tensor 4

real	0m4.002s
user	0m4.000s
sys	0m0.000s
fast 5

real	0m0.856s
user	0m0.850s
sys	0m0.000s
Tensor 5

real	0m5.419s
user	0m5.410s
sys	0m0.000s
fast 6

real	0m5.276s
user	0m5.270s
sys	0m0.010s
Tensor 6

real	0m5.636s
user	0m5.640s
sys	0m0.000s
fast 7

real	0m13.239s
user	0m13.230s
sys	0m0.000s
Tensor 7

real	0m11.848s
user	0m11.840s
sys	0m0.000s
fast 8

real	0m23.668s
user	0m23.660s
sys	0m0.010s
Tensor 8

real	0m19.479s
user	0m19.480s
sys	0m0.000s
fast 9

real	0m34.922s
user	0m34.890s
sys	0m0.000s
Tensor 9

real	0m28.536s
user	0m28.510s
sys	0m0.010s
PASS: one_over_1_minus_x1
PASS: one_over_2_minus_x2
PASS: one_over_3_minus_x3
PASS: one_over_4_minus_x4
PASS: one_over_5_minus_x5
PASS: one_over_6_minus_x6
PASS: one_over_7_minus_x7
PASS: one_over_8_minus_x8
PASS: one_over_9_minus_x9


System setup:
SuSE 8.1
Glibc 2.3.2 
Linux 2.4.19
GNU ld version 2.12.90.0.15 20020717
g++ -v
Reading specs from /usr/local/lib/gcc-lib/i686-pc-linux-gnu/3.3/specs
Configured with: ../gcc3/gcc/configure --enable-threads=posix --enable-languages=c,c++,f77,objc --enable-__cxa_atexit
Thread model: posix
gcc version 3.3 20030314 (prerelease)

Hope this helps,

Peter Schmid