This is the mail archive of the
gcc@gcc.gnu.org
mailing list for the GCC project.
Re: Testsuite policy - test case now passes but bug remains
>>>>> "Martin" == Martin v Loewis <martin@loewis.home.cs.tu-berlin.de> writes:
>> And IMHO, not adding a bug to the testsuite until it is fixed,
>> will insure that it probably takes longer to fix the bug (since
>> it isn't a daily irritant in terms of failing tests). I can't
>> speak for other developers, but for me, it certainly improves
>> the chance I will look at said bug.
Martin> How is that? I thought there was a policy that there
Martin> should be no failing tests, unless they are expected
Martin> failures.
Folks --
We're redebating this in circles. There are plusses and minuses on
all these issues.
As a community, it's taking us a while to learn that the bug
database and regression test suite are two different things. That's
OK; we're getting there.
It's true that FAILs incite bug-fixing. It's true that FAILs
confuse developers. It's true that test-cases appearing only in GNATS
are harder to run. It's true that test-cases in the testsuite tend to
get duplicated. Everybody's right!
Here's what we should do:
If you find a bug, but you're not fixing it:
- Create a test-case.
- Check in the test-case, marking it as XFAIL.
- Write a bug report, referencing the test-case.
- Put the bug report in GNATS.
If you later fix a bug for which there is already a GNATS entry:
- Remove the XFAIL on the test-case.
- Close the bug report in GNATS.
If you find a bug, and you're fixing it right then:
- Create a test-case.
- Check in the test-case, marking it as PASS.
- Check in your fixes.
Let's try this policy out. We can certainly reevaluate it after GCC
3.0. I'm by no means wedded to it.
I think the most important thing is that we have a clear policy.
And that the policy encourages us to use GNATS appropriately, rather
than trying to keep track of bugs by making testsuite entries.
Thanks,
--
Mark Mitchell mark@codesourcery.com
CodeSourcery, LLC http://www.codesourcery.com