Re: Power of the G++ test suite

[Joe Buck <jbuck at synopsys dot com>]

> I've just been having a look at the Plum Hall web site. They
> make commercial C++ test suites.
> 
> They make the claim there that the 1995 version of C++, without its
> library, requires 3000 test cases to test for minimal language conformance.
> They say their language test suite has 4700 tests.

I'm not sure how such a claim can be justified (that you need a specific
N).  I also don't know what they mean by "the 1995 version of C++".  Are
they referring to draft standard documents from around that date?  I think
that the specs were incoherent in several respects around then -- but I
suppose that their test suite could have accepted any of several
interpretations.

> Given that the language has moved on since 1995, and Plum Hall don't say
> anything about how many test cases are required to test the library,
> then can anyone at Cygnus, or anywhere else, comment on the power of
> the G++ test suite, as it currently stands ?

It's too weak to assume that you're getting "minimal coverage".  Also,
because of the way it grew, it has many redundant tests.

> It would seem to me, from Plum Hall's web site, and judging the
> size of the ARM and the C++ working paper, that something possibly over
> 6,000 test cases are required before language conformance can
> be successfully claimed.

>From this statement it appears you are assuming that "1995 version of C++"
== "ARM".  But the ARM is much older than that (1990).  Since Jeff
mentioned that Plum Hall covers namespaces, it appears that their 4700
tests are intended to cover the current version of the language, not the
ARM.