Testsuites

[Andreas Jaeger]

>>>>> Mark Mitchell writes:

 > Folks --

 >   In the past, we've used our testsuites as a combination of testsuite
 > and bug database.

 >   The testsuites in GCC are regression test suites.  We now have a
 > nice GNATS bug-tracking database.

 >   Let's check in new failing tests as XFAILs.  The ideal would be that
 > a build of checked-in sources never causes any FAILs -- only XFAILs.
 > Then, checking that you haven't introduced new bugs is as simple as
 > noticing that there's no test output.  Otherwise, people have to
 > remember the "magic" list of "normal" failures.  That's hard.

 >   For instance, I just updated my tree, in preparation for checking in
 > a change.  Rerunning the tests, I suddenly saw failures.  It required
 > analysis on my part to figure out that the new failures stemmed from
 > new tests, not from bugs in my code.

 >   Bug reports should go in GNATS.  The maintainer for that part of the
 > code should mark it high priority, so that we know we have to fix it
 > before the next release.  But the test-case should still be XFAIL --
 > it is, after all, expected to fail.

What should we do with existing FAILs?  Shouldn't we mark all existing
FAILs as XFAILs now?  This has already happened for c++ but not for
the other languages.

Andreas
-- 
 Andreas Jaeger
  SuSE Labs aj@suse.de
   private aj@arthur.inka.de