A testcase library

[David Carlton]

On Mon, 23 Dec 02 20:45:36 EST, kenner@vlsi1.ultra.nyu.edu (Richard Kenner) said:

>> Suppose that GDB is debugging a program, and the information it
>> needs is simply missing from the debug info.  Zilch, nada,
>> absolutely no way to recover it.  That's an XFAIL.

>> Suppose that a testcase is for a new bug that no one's figured out
>> how to fix yet.  That's a KFAIL.  All KFAILs are required to have
>> an open PR associated with them, at least in GDB-land.

>> The difference, in theory, is that KFAILs represent real problems
>> that have not been fixed; and XFAILs represent "expected" failures,
>> problems in the system or tools that can not be fixed in the
>> program-under-test.

> I still don't get it.  Your first example sounds like something
> that's not a bug at all, in which case why would the "test case"
> still be in the suite?

If something XFAILs on all systems, then it would be a little strange
to have it in the test suite.  But a more typical situation is that
there's something that GDB tries to get right, and can on certain
systems, but can't on other systems; so we XFAIL it where we can't get
it right.

For example, say that I'm doing a bunch of tests for GDB's namespace
support.  Then there might be some areas of namespace support that GDB
doesn't try to handle at all; for those, I submit PR's, and add
KFAILed tests.

Then there might be some tests that, say, GDB gets right if we have
DWARF-2 debug info but not with stabs, because stabs doesn't have good
enough nesting constructs.  I'll add tests for those to the test
suite, but I'll XFAIL them if the current debug format is stabs.

We'll see how it turns out: we added the first KFAIL to GDB's test
suite all of a week ago, so it's not yet clear exactly how the extra
information will help.  Probably its main benefit won't be in that
extra information, but will rather be that it gives people an excuse
to spend time auditing current FAILs and XFAILs (and XPASSes, for that
matter), with the result that the bug database will more closely
reflect GDB's actual state.  (Though that only helps if it in turn
encourages people to fix bugs...)

David Carlton
carlton@math.stanford.edu