Re: Power of the G++ test suite

[mrs at wrs dot com (Mike Stump)]

> Date: Fri, 17 Jul 1998 20:42:00 +0100 (BST)
> From: David C Binderman <dcb@pncl.co.uk>

> can anyone at Cygnus, or anywhere else, comment on the power of the
> G++ test suite, as it currently stands ?

Imagine that you took a couple hundred millions lines of code and ran
it through g++ at various times over the past decade.  Imagine that
you collected the things that failed in various ways with g++, and put
that into a testsuite.  Well, that is a reasonable way to think about
the g++ testsuite.  They are mostly things that used to fail in g++
and things that we want to ensure stay fixed.

In some respects this is more valuable than a commercial testsuite, as
it tests those things where the internal code was (or is) weak or
confusing.  It also tests those areas were there has been a bit of
code motion internally, or reorganizations in code generation
strategies, while the commercial testsuites are quite as tailored to
g++.

But that is also a limitation of the g++ testsuite.  It was developed
along with g++ at roughly the same time and by roughly the same
people.  The commercial testsuites are good because they are
disconnected from g++, because other compiler vendors use them and
because they represent a different set of eyes reading the words in
the standard.

The g++ testsuite is good at ensuring a minimum quality of the
compiler.  It allows people to consider reimplementing parts of the
compiler knowing that if they mess it up, the testsuite is likely to
complain (if they listen to it).  Also, it enables a machine porter to
port the compiler, and run the test, and be reasonably confident that
they can claim g++ works, without knowing C++ or anyhting about g++ in
particular. They are less likely to do this with a commercial
testsuite as they would have to be serious enough about C++ to pay out
for a commercial testsuite ($xx,xxx).  Because the g++ testsuite is
free, anyone or is even remotely curious can obtain it, and test g++.
This translates into a well tested compiler.