Mon Jun 5 02:52:00 GMT 2000
At 10:56 04.06.00, Mark Mitchell wrote:
> In the past, we've used our testsuites as a combination of testsuite
>and bug database.
> The testsuites in GCC are regression test suites. We now have a
>nice GNATS bug-tracking database.
> Let's check in new failing tests as XFAILs. The ideal would be that
>a build of checked-in sources never causes any FAILs -- only XFAILs.
>Then, checking that you haven't introduced new bugs is as simple as
>noticing that there's no test output. Otherwise, people have to
>remember the "magic" list of "normal" failures. That's hard.
> For instance, I just updated my tree, in preparation for checking in
>a change. Rerunning the tests, I suddenly saw failures. It required
>analysis on my part to figure out that the new failures stemmed from
>new tests, not from bugs in my code.
Maybe I'm alone, but I don't think this is a good idea with the current
setup, at least with the C tests. It happens that I have some time to spend
and I try to backport fixes from the mainline to the branch. To see what I
might tackle, I run the mainline testsuite with the branch compiler and
look at the FAILs, this won't be easy anymore, because they maybe XFAILs
now, or? How can I find out easily which tests are really FAILs and which
XFAILs? Even now it's extremely hard to link bug posting, testcase and bug
fix together (partly because the web search engine can't be told to search
for "20000506-1.c" as a single word). I know this will get better with
continued use of GNATS, but right now...
These are the problems with the current testsuite as I see them:
- it's sometimes near to impossible to tell for an outsider what a XFAIL on
a testcase means:
* testcase should fail (a FAIL is the correct
* testcase XFAILs now, but is a bug and should be tackled sometime
* testcase XFAILs with an ICE, I really don't understand the
reasoning behind that... This doesn't distinguish between C++ internal
numbered ICE's and enable-checking ICE's, which might even hide an easy
bugfix in my eyes.
- the C++ testsuite doesn't loop over optimization options, an additional
run with -O2 would be nice here, this would also let the testcases override
the optimization correctly.
A possible solution in my eyes would be 2 additional categories, maybe like
XBUG (expected fail, but real bug) and XICE (to mark expected ICEs). XICE
shouldn't filter out enable-checking ICE's though...
What do you think?
More information about the Gcc