GCC testing questions

[Robert Lipe]

> > > From: gary@Intrepid.Com (Gary Funck) 
> > > We're doing some work with an older 2.7.2 based GCC
[ ... ]
> > > The client using the compiler asked how the compiler is tested,
>
> Thanks, Mike.  Those are good suggestions.  It won't be possible
> moving to the latest GCC because the current build of the product
> passes its tests, and going with new tools would potentially increase
> risk, and extend

The hardball answer is that if changing the compiler introduces risk of
product failure then your testing is inadequate. :-)

> Your answer spoke to what level of passing might be considered
> adequate (ie, less than 10 failures), but not exactly what standard
> is used by the GCC community to determine when the base compiler is
> "done" and is ready for deployment on the major platforms (what _are_
> those major platforms? <g>).

The answers were different in the 2.7 era.  (I think they were much
more fuzzy then, too.)  

Even for the modern releases, since the "acceptable" failures vary
from target to target, there isn't a fixed answer.  It's the reality
of a multi-architecture compiler like GCC that different failures
will show on different targets.  We go through this exercise on major
releases.  The last major release of GCC was 2.95.  Reference:
http://gcc.gnu.org/gcc-2.95/gcc-2.95.2.html .

Follow the link to 'regress.html'.  That shows the list of targets that
were considered major at the time.  It was a goal that the last release
(1.1.2) would have the 2.95 testsuite run on all the major targets and
that the goal was to have zero regressions from 1.1.2.  (It looks like
one of my targets was the only one to blemish that shiny goal.)  There
are probably a few hundred thousand cominbations exercised.

There are also links showing which architectures were reported to work
even they didn't run the testsuite and sign up for the "no regressions"
target.

The big-name testsuites carry serious licensing costs and restrictions.
Therefore the public releases aren't formally certified as passing
those.  However, there are companies with licenses to use that do run
those suites and can provide certified GCC releases if it's important
enough to you.  Of course, the better citizens feed back any changes
necessary to pass the suites, so the reality is that the code generally
stays pretty conformant.

RJL