A testcase library

[Jason R Thorpe]

On Fri, Dec 20, 2002 at 02:13:23PM +0100, Volker Reichelt wrote:

 > =================================================================================
 > On Fri, Nov 29, 2002 at 09:47:24AM +0000, Nathan Sidwell wrote:
 > > Craig Rodrigues wrote:
 > > > A few questions:
 > > > (1) Can I commit my testcase to the testsuite, since it is definitely
 > > >     a regression from gcc 3.1 and gcc 3.2?
 > > No, we only commit test cases when the bug's fixed. The rationale being you
 > > don't want to add a new failing test case -- that'd confuse people. Marking
 > > it XFAIL isn't really the right solution, as then when you fix the bug, you
 > > don't know whether fixing the XFAIL was accidental, and end up adding a new
 > > test case anyway. Also, in fixing the bug, you (well I at least), can
 > > construct a better test case having undersytood the failure mode.
 > =================================================================================

I have to say that I disagree with this policy, esp. if the testcase shows
a regression from a previous GCC release.  Gaps in the testsuite mean it
will take longer for bugs to be fixed, and give people a false sense of
confidence in the quality of the compiler when they see results that show
few, if any, failures.

One example is a PR I filed recently, with test case that causes the compiler
to loop forever.  The rationale I got for not adding the testcase was "it
would make everyone's bootstrap/test cycle 5 minutes longer".  My initial
thought was "...and that will motivate someone with the knowledge to do so
to fix it." (The test case in question shows a regression from a previous
release of GCC.)

...the principle is not too different than a suggestion someone made a
while ago about making GCC maintainers do bootstraps on 100MHz Pentiums
(maybe GCC wouldn't slow down so much over time if they did :-)

-- 
        -- Jason R. Thorpe <thorpej@wasabisystems.com>