Notes from the testing BOF at the summit

[Ben Elliston <bje at au dot ibm dot com>]

We held a testing BOF at the summit yesterday.  It was a short affair,
as we had very limited time, but I think we managed to cover the most
important issues on peoples' minds.  I took notes, which I posting
here.

The first issue, which I raised, is that Dejagnu is now a more
actively maintained project in a repository on savannah.gnu.org and is
duplicated in the sources.redhat.com "src" repository.  This is a
growing problem because there have been instances where these versions
of Dejagnu have not been in sync--some developers have committed
Dejagnu patches to src/dejagnu and not sent them upstream.  Since
Dejagnu is now maintained--and there is a 1.4 maintenance branch from
which urgent bug fix releases can be made--the consensus was that
src/dejagnu can be removed.

Expect is also independently maintained, yet an old (and potentially
patched) version is carried in src/expect.  This, too, can be removed,
however it would be a good idea to review all local patches made to
Expect since its initial import into sources.redhat.com and send those
patches upstream to Don Libes.

Once this code has been flushed, developers will have to install
Dejagnu and Expect themselves.  Many distributions and packaging
systems now carry the correct versions.  In the case of GCC, we need
to ensure that the requirements for testing are clearly stated, if
they are not already.

The consistency in interpretation of test result state semantics is a
known problem.  For instance, GDB uses Dejagnu's new KFAIL state for
"known failures", whereas the G++ testsuite uses XFAIL.  The semantics
need to be clarified in the manual and these inconsistencies resolved
between different GNU tools.

Some requested enhancements were discussed.  These include:

  * The ability for dg.exp tests to have test outcomes such as 
    UNTESTED, UNSUPPORTED and UNRESOLVED;

  * Dejagnu should prefix the test output with the name of the test
    case to help differentiate tests with similar or identical test
    strings;

  * Parallel testing.  The GCC and GDB testsuites already have the
    ability to test multilibs in parallel using a tricky target
    contributed by Alexandre Oliva, but the ability to run individual
    tests within, say, compile.exp, is considered essential.

  * The documentation needs to better spell out how to run individual
    tests.

The idea of requiring more coverage from new test cases (perhaps with
a coverage report included with patches submitted) was discussed, but
no real consensus was reached.  Ditto for the idea of creating a
collection of unit tests (for which code such as real.c would be a
good candidate).

Any comments?

Ben