Re: Testsuites

["Martin v. Loewis" <martin at loewis dot home dot cs dot tu-berlin dot de>]

> Maybe I'm alone, but I don't think this is a good idea with the
> current setup, at least with the C tests. It happens that I have
> some time to spend and I try to backport fixes from the mainline to
> the branch. To see what I might tackle, I run the mainline testsuite
> with the branch compiler and look at the FAILs, this won't be easy
> anymore, because they maybe XFAILs now, or?

To compare test suite results between the mainline and the release
branch, I'd recommend using the same test suite for both compilers -
comparing the different test suites is quite unreliable, as many tests
have been changed since the branch was created.

>          * testcase should fail (a FAIL is the correct 
> compilation/execution response)

How many of those are there? That really should *never* happen; in the
only case where it does, it indicates a deficiency in dejagnu (AFAIK).

>          * testcase XFAILs now, but is a bug and should be tackled sometime
>          * testcase XFAILs with an ICE

What is the difference between those two? An ICE is a bug is an ICE.

> I really don't understand the reasoning behind that... This doesn't
> distinguish between C++ internal numbered ICE's and enable-checking
> ICE's, which might even hide an easy bugfix in my eyes.

Not all ICEs are easy to fix, though - I have a long list of more than
hundred pending ICE reports, and I certainly don't want to show those
up as FAIL - some of them may not get fixed before GCC 4.

> What do you think?

I think Mark's proposal is very good, and all test cases should PASS
or XFAIL by default; any other result indicates that some patch should
not have been installed.

Regards,
Martin