Re: QMTest and the G++ testsuite

["Joseph S. Myers" <jsm28 at cam dot ac dot uk>]

On Mon, 20 May 2002, Mark Mitchell wrote:

> 2. Support for direct comparisons between test results.  QMTest can
>    easily answer the question "Did I break anything relative to the
>    results I have from yesterday?"

How?  This doesn't seem to be covered in the README.  What's the
equivalent of running diff against saved .sum files?

> I've attached the file README.QMTEST, now present in the "testsuite"
> directory, that explains how to try out "make qmtest-g++" instead of
> "make check-g++".

When this is nonexperimental (i.e., when QMTest is the recommended or only
way to run tests), could the docs please go in the internals manual
(sourcebuild.texi) rather than having yet more miscellaneous READMEs about
the place?  There are some limited test suite docs there, describing the
structure of the C testsuite directories and some of the idioms used in
test cases (including one used just because of limitations of DejaGnu -
testing for where a message should be a hard error, not a warning); a lot
more documentation is needed.

Should people working on testsuites intended to go in GCC at some later 
date (e.g. ACATS, the GNU Pascal testsuite) now work towards QMTest 
harnesses for them, rather than DejaGnu ones?

>   With QMTest, each source file is considered a single test.  If any
>   of the seven sub-tests fail, the entire test is considered to fail.
>   However, QMTest does present information about *why* the test
>   failed, so the same information is effectively available.
> 
>   It is true that, therefore, causing an already failing test to "fail
>   more" is not immediately detectable through an additional unexpected
>   failure messages when using QMTest.  On the other hand, most people
>   seem to think of each source file as "a test", not "twelve tests",
>   so the model QMTest uses may be more natural.

How does this work where a single file contains both tests expected to 
pass and those expected to fail?  (That is, in such a case, what results 
does QMTest give if all tests are as expected, or if a test meant to work 
is broken, or if a test expected to fail starts passing?)  (For example, 
gcc.dg/sequence-pt-1.c tests both cases that are implemented, and cases 
that GCC in principle could warn for but doesn't at present.)

-- 
Joseph S. Myers
jsm28@cam.ac.uk