Testsuites

[Mark Mitchell <mark at codesourcery dot com>]

Folks --

  In the past, we've used our testsuites as a combination of testsuite
and bug database.

  The testsuites in GCC are regression test suites.  We now have a
nice GNATS bug-tracking database.

  Let's check in new failing tests as XFAILs.  The ideal would be that
a build of checked-in sources never causes any FAILs -- only XFAILs.
Then, checking that you haven't introduced new bugs is as simple as
noticing that there's no test output.  Otherwise, people have to
remember the "magic" list of "normal" failures.  That's hard.

  For instance, I just updated my tree, in preparation for checking in
a change.  Rerunning the tests, I suddenly saw failures.  It required
analysis on my part to figure out that the new failures stemmed from
new tests, not from bugs in my code.

  Bug reports should go in GNATS.  The maintainer for that part of the
code should mark it high priority, so that we know we have to fix it
before the next release.  But the test-case should still be XFAIL --
it is, after all, expected to fail.

  Thanks,

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com