Re: [testsuite] C++ i386 ICE

[Franz Sirl <Franz dot Sirl-kernel at lauterbach dot com>]

At 15:50 14.06.00, Jeffrey A Law wrote:
>   In message <39474516.DED75402@codesourcery.com>you write:
>   > Jason Merrill wrote:
>   > >
>   > > >>>>> Jeffrey A Law <law@cygnus.com> writes:
>   > >
>   > >  > This is fine.  THough the C++ folks might want it marked as xfail.
>   > >
>   > > I'd like to revise that policy slightly; tests which work with past
>   > > releases should not be marked as XFAIL.  Any objections?
>   > would it not be better to put them in GNATS as critical severity?
>That's fine too, but I really like the capability of being able to run
>the testsuite and get a list of failures that we would consider regressions
>for a release.

I was thinking about this testsuite stuff quite a lot during the last two 
weeks (inspired by a related discussion with Mark), and I have come up with 
this suggested procedure:

Preparations:
1. re-evaluate all current XFAILs in all testsuites if they are real bugs 
or used to workaround deficiences of the testsuite
2. change "make check" and the test_summary script to list FAIL, XFAIL and 
XPASS cases, that way all bugs are exposed to the occasional 
user/contributor and to the gcc-testresults mailing list
3. introduce a new "make check-maintainer" that hides XFAIL, like the 
current "make check" does, that way

Procedures:
1. checking in new testcases (or remove wrong testcases) _requires_ a 
reference to a GNATS bug database entry or a link to the mailing list 
archive in the testsuite ChangeLog and/or cvs log.
2. unless the contributor exactly knows the status of the testcase, check 
it in as FAIL
3. after the smoke has cleared and discussion on the lists, downgrade the 
testcase to XFAIL (unless it has been fixed :-) )
4. if a release is pending, decide which bugs should be fixed for the 
release, and upgrade them to FAIL in the release-branch

What do ya think?
Franz.