Questions about the gcc testsuite

[Ryan Smiderle]

I guess that's why some people are pushing for using the QMTest framework?
The method for actually setting/removing xfails in the testsuite is quite
a hassle.  I was thinking of setting up something similar to what you have..
Do you just allow dejagnu to report failures, then run the log file through
another program that marks them xfails?

-Ryan

> 
> On Tuesday, June 10, 2003, at 12:09 PM, Eric Christopher wrote:
> > On Tue, 2003-06-10 at 12:09, Ryan Smiderle wrote:
> >> Ok, thanks.. just making sure.
> >>
> >> Also.. when a version of GCC is released, is the testsuite completely
> >> up to date as far as setting up xfails goes?  So that when the 
> >> testsuite
> >> is ran, you get a bunch of XFAILS, but zero FAILS?
> >
> > Nope. This issue has been raised and various people are of different
> > minds about it :)
> >
> > Personally I think we should probably xfail our release branches, but
> > not mainline. Anyone else have any thoughts here?
> 
> On primary platforms, we should mandate test results for release 
> candidates, and that the results contain 0 FAILs for all testsuites for 
> a release.
> 
> If people don't want to test releases or mark the FAILs as ok to ship 
> with, then it isn't a primary platform, is it?
> 
> At Apple, we have a scheme somewhat external to dejagnu that let's us 
> mark failures with bug numbers (a closely related feature is mostly 
> available in dejagnu), and for shipment, we mandate 0 in the column 
> that contains FAILs without such bug tracking numbers.  The scheme is 
> nice in many ways, and I think could be applied to the FSF tree and 
> bugzilla.
> 
> Doing this aids developers, as then one could just run the testsuite 
> once before submission, and notice the number of FAILs should be 0.  
> None of this, build it and run the testsuite with and without the patch 
> and compare stuff would be necessary.
> 
> Doing this does create PRs for FAILs on popular platforms, and they can 
> be ignored, fixed or worked around as maintainers have time.  The key 
> point is that it doesn't mandate work for developers, but it does 
> mandate non-complex work for people that file bug reports against the 
> test results.  Little to no compiler experience necessary.
> 
> As an incremental step in getting there, we could mandate just _1_ 
> platform, and try it out.  If it works, we could add more.  Internally, 
> it doesn't seem to be too costly to do.
>