Re: Questions about the gcc testsuite

[Mike Stump <mrs at apple dot com>]

On Tuesday, June 10, 2003, at 12:09 PM, Eric Christopher wrote:
> On Tue, 2003-06-10 at 12:09, Ryan Smiderle wrote:
> > Ok, thanks.. just making sure.
> >
> > Also.. when a version of GCC is released, is the testsuite completely
> > up to date as far as setting up xfails goes?  So that when the 
> > testsuite
> > is ran, you get a bunch of XFAILS, but zero FAILS?
>
> Nope. This issue has been raised and various people are of different
> minds about it :)
>
> Personally I think we should probably xfail our release branches, but
> not mainline. Anyone else have any thoughts here?

On primary platforms, we should mandate test results for release 
candidates, and that the results contain 0 FAILs for all testsuites for 
a release.

If people don't want to test releases or mark the FAILs as ok to ship 
with, then it isn't a primary platform, is it?

At Apple, we have a scheme somewhat external to dejagnu that let's us 
mark failures with bug numbers (a closely related feature is mostly 
available in dejagnu), and for shipment, we mandate 0 in the column 
that contains FAILs without such bug tracking numbers.  The scheme is 
nice in many ways, and I think could be applied to the FSF tree and 
bugzilla.

Doing this aids developers, as then one could just run the testsuite 
once before submission, and notice the number of FAILs should be 0.  
None of this, build it and run the testsuite with and without the patch 
and compare stuff would be necessary.

Doing this does create PRs for FAILs on popular platforms, and they can 
be ignored, fixed or worked around as maintainers have time.  The key 
point is that it doesn't mandate work for developers, but it does 
mandate non-complex work for people that file bug reports against the 
test results.  Little to no compiler experience necessary.

As an incremental step in getting there, we could mandate just _1_ 
platform, and try it out.  If it works, we could add more.  Internally, 
it doesn't seem to be too costly to do.