Suggestion for a new GNATS policy

[Giovanni Bajo]

Joseph S. Myers <jsm28@cam.ac.uk> wrote:

> I'll reiterate that if we put these testcases in the testsuite, XFAILed
> and with cross-references from testcase to bug number and from bug to
> testcase, they would be automatically tested by many people on many
> platforms and an accidental fix would be noticed immediately as an XPASS
> when the patch accidentally fixing the bug is tested.  In that case the
> keyword "monitored" could be used to mean "there is an XFAILed testcase in
> the testsuite for this bug, and it isn't specific to a rarely tested
> platform".

My only problem with this is that we would end up cluttering the testsuite
with testcases which are not really bugs or are not completely correct, due
to human mistakes / misinterpretations. Adding the testcase to the testsuite
only when the bug is really fixed makes it sure that it's been "approved" as
a real bug.
In other words, XFAIL's meaning could be silently overloaded with "this is
failing, but after all we're not 100% sure that this is a real bug".
Of course, this applies to C/C++ standard interpretation. An ICE is of
course always a bug.

In the end, I think that using the testsuite would not cover correctly _all_
the PRs. Thus, some would have to still be manually checked. Hence, we can
do that for everything and leave the testsuite "pure".

Giovanni Bajo