Re: Testsuites

[Franz Sirl <Franz dot Sirl-kernel at lauterbach dot com>]

At 10:56 04.06.00, Mark Mitchell wrote:

>Folks --
>
>   In the past, we've used our testsuites as a combination of testsuite
>and bug database.
>
>   The testsuites in GCC are regression test suites.  We now have a
>nice GNATS bug-tracking database.
>
>   Let's check in new failing tests as XFAILs.  The ideal would be that
>a build of checked-in sources never causes any FAILs -- only XFAILs.
>Then, checking that you haven't introduced new bugs is as simple as
>noticing that there's no test output.  Otherwise, people have to
>remember the "magic" list of "normal" failures.  That's hard.
>
>   For instance, I just updated my tree, in preparation for checking in
>a change.  Rerunning the tests, I suddenly saw failures.  It required
>analysis on my part to figure out that the new failures stemmed from
>new tests, not from bugs in my code.

Maybe I'm alone, but I don't think this is a good idea with the current 
setup, at least with the C tests. It happens that I have some time to spend 
and I try to backport fixes from the mainline to the branch. To see what I 
might tackle, I run the mainline testsuite with the branch compiler and 
look at the FAILs, this won't be easy anymore, because they maybe XFAILs 
now, or? How can I find out easily which tests are really FAILs and which 
XFAILs? Even now it's extremely hard to link bug posting, testcase and bug 
fix together (partly because the web search engine can't be told to search 
for "20000506-1.c" as a single word). I know this will get better with 
continued use of GNATS, but right now...

These are the problems with the current testsuite as I see them:
- it's sometimes near to impossible to tell for an outsider what a XFAIL on 
a testcase means:
         * testcase should fail (a FAIL is the correct 
compilation/execution response)
         * testcase XFAILs now, but is a bug and should be tackled sometime
         * testcase XFAILs with an ICE, I really don't understand the 
reasoning behind that... This doesn't distinguish between C++ internal 
numbered ICE's and enable-checking ICE's, which might even hide an easy 
bugfix in my eyes.
- the C++ testsuite doesn't loop over optimization options, an additional 
run with -O2 would be nice here, this would also let the testcases override 
the optimization correctly.

A possible solution in my eyes would be 2 additional categories, maybe like 
XBUG (expected fail, but real bug) and XICE (to mark expected ICEs). XICE 
shouldn't filter out enable-checking ICE's though...

What do you think?

Franz.