Re: GCC 3.0 Release Criteria

[Jonathan Thornburg <jthorn at galileo dot thp dot univie dot ac dot at>]

A couple of comments/suggestions:

> Applications
> 
>    In addition to the regression-testing mentioned above, it is important
>    that operation of the compiler be verified on some real-word
>    applications. The following applications represent a mix of low-level
>    and high-level code, of numerical and logical programs, and of
>    different programming languages.
>    
>                          CAPTION: Integration Tests
>                                       
>                       Name Language Version Source URL
>                  Linux kernel C 2.2.14 linux-2.2.14.tar.gz
>                      GNU Emacs C 20.6 emacs-20.6.tar.gz
>                       POOMA C++ 2.2.0 pooma-2.2.0.tgz
>                              LAPACK Fortran 2.0
                                              ===

I'm surprised to see LAPACK 2.0 (dated 30 Sept 1994 according to
the Netlib web site) specified, when LAPACK 3.0update is available
(dated 31 Oct 1999).  Is there a deep reason for this?

Since LAPACK performance depends heavily on BLAS performance
(a lower-level subroutine library called by LAPACK), perhaps the
release criteria should say explicitly what BLAS versions will be used?
I would suggest either the reference implementation (very vanilla Fortran),
or (probably more realistic) the Kagstrom, Ling, and Van Loan GEMM-based
implementation (still portable Fortran, but tuned for computers with
caches).  Both are available from the same netlib sites that distribute
LAPACK.



>   These selections were made for a variety of reasons. The Linux kernel
>   [[...]]
>   
>   GNU Emacs [[...]]
>   
>   POOMA [[...]]

Right now there's no mention of what LAPACK is and why it was chosen.
Perhaps a sentence or two should be added, perhaps along the lines of

   LAPACK is a large subroutine library for numerical linear algebra.
   It's available in Fortran 77, Fortran 90, and C++ versions.
   Particularly in its Fortran 77 version, it's representative of
   much "number-crunching" code, and should particularly stress GCC's
   loop optimizations.



>Therefore, we will use the following benchmarks for
>    measuring code quality:
>    
>                           Name Language Source URL
>                          gzip C gzip-1.2.4a.tar.gz
>                         Stepanov C++ stepanov_v1p2.C
>                                   Fortran
>                                       
>    A Java benchmark is not required for this release since there is
>    little precedent for the behavior of the Java compiler. For Java,
>    functional completeness and correctness are still more important than
>    optimization.
>    In addition to the above benchmarks, the behavior of real programs
>    should be considered as well. For that reason, the behavior of the
>    elliptic curve integer factorization program ecm4c.c, which uses GNU
>    mp, will be considered part of the release criteria.

Why is the set of code-quality benchmarks different from the set of
application benchmarks?  Particularly since some of the applications
(POOMA, LAPACK) are cpu-intensive, why not also use them for code-quality
criteria as well?



> Compile-Time Performance
> 
>    In order to measure compile-time performance, we will use the
>    following unit tests:
>    
>    Name Language Source Flags Comments
>    insn-attrtab.c C -O2
>[[...]]
>    In addition to these unit tests, we will measure the time and peak
>    memory usage used when building the entire GNU Emacs distribution with
>    both GCC 2.95.2 and GCC 3.0.

Same idea as before -- POOMA would certainly be a good stress test
for compile-time performance on C++, and perhaps LAPACK for Fortran?



One other idea... if it weren't for the problems of non-free test code,
I think the SPEC CPU2000 benchmark suite would be a *great* set of
application tests.  However, many testers won't have access to the SPEC
code (which costs money), so this may not be practical.

-- 
-- Jonathan Thornburg <jthorn@galileo.thp.univie.ac.at>
   http://www.thp.univie.ac.at/~jthorn/home.html
   Universitaet Wien (Vienna, Austria) / Institut fuer Theoretische Physik
   "Stock prices have reached what looks like a permanently high plateau"
   -- noted economist Irving Fisher, 15 October 1929