GCC options and floating-point correctness (benchmarks)

[Scott Robert Ladd <coyote at coyotegulch dot com>]

Hello,

One of the most prominent numerical programmers, Dr. William Kahan, 
developed the PARANOIA benchmark. While it doesn't cover every aspect of 
numerical "correctness", it does provide an interesting benchmakr for 
comparing the effects of different compiler options.

Among numerical programmers, Mr. Kahan holds a reputation similar to 
that of Donald Knuth.

I produced the following results for a C version of PARANOIA on a 2.8GHz 
Pentium 4 (Northwood core), HT enabled, using a very recent build of the 
tree-ssa branch. I'll be glad to send copies of the benchmark source to 
anyone who requests it privately.

Conclusions and observations:

The best results (and the only potentially acceptable one in Mr. Kahan's 
opinion) required the use of the -march=pentium4 and -mfpmath=sse options.

VERY IMPORTANT: The -ffast-math switch *DOES NOT* introduce failures and 
defects *UNLESS COMBINED* with one of -O1, -O2, or -O3. And yes, 
-ffast-math does alter code generation even if no optimization is specified.

In fact, the use of any optimizations (-O1, -O2, or -O3) introduces 
numerous failures and defects (with and without -ffast-math), *unless* 
those switches are combined with -march=pentium4 and -mfpmath=sse. It is 
the -O options that should be our concern, *NOT* -ffast-math.

Adding -ffast-math to -O3 (without any other code generation options) 
actually *reduces* the number of failures over a simple -O3.

gcc -o paranoiag -lm paranoia.c

(no failures!)
The number of  SERIOUS DEFECTs  discovered = 1.
The number of  DEFECTs  discovered =         1.
The number of  FLAWs  discovered =           1.

gcc -o paranoiag -lm -march=pentium4 -mfpmath=sse paranoia.c
gcc -o paranoiag -lm -O1 -march=pentium4 -mfpmath=sse paranoia.c
gcc -o paranoiag -lm -O2 -march=pentium4 -mfpmath=sse paranoia.c
gcc -o paranoiag -lm -O3 -march=pentium4 -mfpmath=sse paranoia.c
gcc -o paranoiag -lm -ffast-math -march=pentium4 -mfpmath=sse paranoia.c

(no failures)
(no serious defects)
The number of  DEFECTs  discovered =         1.
(no flaws)

gcc -o paranoiag -lm -O1 paranoia.c

The number of  FAILUREs  encountered =       3.
The number of  SERIOUS DEFECTs  discovered = 3.
The number of  DEFECTs  discovered =         5.
The number of  FLAWs  discovered =           2.

gcc -o paranoiag -lm -O2 paranoia.c

The number of  FAILUREs  encountered =       3.
The number of  SERIOUS DEFECTs  discovered = 3.
The number of  DEFECTs  discovered =         5.
The number of  FLAWs  discovered =           2.

gcc -o paranoiag -lm -O3 paranoia.c

The number of  FAILUREs  encountered =       4.
The number of  SERIOUS DEFECTs  discovered = 4.
The number of  DEFECTs  discovered =         3.
The number of  FLAWs  discovered =           2.

gcc -o paranoiag -lm -O3 -ffast-math paranoia.c

The number of  FAILUREs  encountered =       3.
The number of  SERIOUS DEFECTs  discovered = 4.
The number of  DEFECTs  discovered =         3.
The number of  FLAWs  discovered =           2.

gcc -o paranoiag -lm -O3 -ffast-math -march=pentium4 paranoia.c

The number of  FAILUREs  encountered =       3.
The number of  SERIOUS DEFECTs  discovered = 3.
The number of  DEFECTs  discovered =         3.
The number of  FLAWs  discovered =           2.

gcc -o paranoiag -lm -O3 -ffast-math -march=pentium4 -mfpmath=sse paranoia.c

The number of  SERIOUS DEFECTs  discovered = 1.
The number of  DEFECTs  discovered =         1.
The number of  FLAWs  discovered =           2.

--
Scott Robert Ladd
Coyote Gulch Productions (http://www.coyotegulch.com)
Software Invention for High-Performance Computing