This is the mail archive of the
gcc@gcc.gnu.org
mailing list for the GCC project.
Re: Suggestion for a new GNATS policy
On Mon, May 12, 2003 at 04:08:13AM +0200, Giovanni Bajo wrote:
> Joseph S. Myers <jsm28@cam.ac.uk> wrote:
>
> > I'll reiterate that if we put these testcases in the testsuite, XFAILed
> > and with cross-references from testcase to bug number and from bug to
> > testcase, they would be automatically tested by many people on many
> > platforms and an accidental fix would be noticed immediately as an XPASS
> > when the patch accidentally fixing the bug is tested. In that case the
> > keyword "monitored" could be used to mean "there is an XFAILed testcase in
> > the testsuite for this bug, and it isn't specific to a rarely tested
> > platform".
>
> My only problem with this is that we would end up cluttering the testsuite
> with testcases which are not really bugs or are not completely correct, due
> to human mistakes / misinterpretations. Adding the testcase to the testsuite
> only when the bug is really fixed makes it sure that it's been "approved" as
> a real bug.
> In other words, XFAIL's meaning could be silently overloaded with "this is
> failing, but after all we're not 100% sure that this is a real bug".
> Of course, this applies to C/C++ standard interpretation. An ICE is of
> course always a bug.
>
> In the end, I think that using the testsuite would not cover correctly _all_
> the PRs. Thus, some would have to still be manually checked. Hence, we can
> do that for everything and leave the testsuite "pure".
While I agree that we don't want to clutter up the testsuite, if these
tests were run regularly on a variety of platforms then we could easily
identify which platforms are affected and notice when a bug is magically
fixed. Perhaps the PR tests could be in a separate testsuite that is
easy to plug into regular test runs (as with Mauve, for example) and
whose results could be reported in parallel to those of the regular
testsuite. It would need to be easy for the people analyzing PRs to add
new tests to this suite.
Janis