Re: Power of the G++ test suite

[Joe Buck <jbuck at synopsys dot com>]

> I'm surprised. If the ARM has N sentences in it and the C++ final
> spec has M sentences, then I would expect that a test suite claiming 
> ARM conformance would have at least N tests in it, and one claiming
> C++ final standard conformance to have at least M tests in it.

That's probably a reasonable estimate.  The sentences may not be
independent, or one sentence may require many tests.  For example,
how many tests would you need to verify this statement:

-----------------------------
1 A member of a class can be
  ...
  --protected; that is, its name can be used only by  member  functions,
    static  data  members,  and  friends  of  the  class  in which it is
    declared and by member functions, static data members,  and  friends
    of classes derived from this class.
    (see _<A HREF="#class.protected">class.protected</A>_).

-----------------------------

g++ has a history of granting access to protected members when it
shouldn't.  But a test suite can only try a finite number of attempts to
illegally access protected members.  I suspect that by now, the g++ test
suite has more tests of protected member access than Plum Hall does,
simply because of the way it was built (from user bug reports).  Still,
every once in a while until quite recently new problems have been found.

> I still think that trying to claim that the current G++ test suite is 
> anything like comprehensive, or complete, is wrong.

No one has made such a claim.  (I suppose Mike Stump came closest to using
language that suggested such a claim, but all he really said is that it's
a reasonable check to see if a port to a new platform is mostly correct).

> Hence any claim that G++ has been extensively tested, solely on the
> basis of having run the G++ test suite over it, cannot, IMHO, be true.

Right, but again no one has made such a claim.

You have ignored statements from Cygnus folks that they have licensed the
Plum Hall test suite and have used it to test g++.  They cannot legally
say how well g++ did, thanks to Plum Hall's license.  (It seems Plum Hall
wants you to shell out $20K if you want to know).