Re: QMTest Update

[Jim Wilson <wilson at redhat dot com>]

I finally got around to trying QMTest.

The docs say that Python 2.1 (or greater) is required.  So I tried the version
the docs recommended (2.1.3) and discovered that it doesn't work.  QMTest is
using a python function (PyType_IsSubtype) that only exists in Python 2.2.
The docs in qmtest and gcc need to be fixed.

Testcase results get reported in a non-deterministic order.  This is fine for
batch testing and parallel testing, but inconvenient for real-time testing.
If I am testing on a slow target that is going to take hours or even days to
finish, then I'd like to be able to spot regressions as they appear, instead of
being forced to wait until testing is done.  Dejagnu runs tests in alphabetical
order, making it easy to compare results by hand against a previous run as
they come out.

QMTest is leaving behind the final file for many testcases.  Depending on the
testcase, this can be a .s file, or a .o file, or a .exe file.  Most of these
files don't appear to be of any use.  Dejagnu only leaves a file behind if a
testcase fails, so that one can try looking at it to see why it failed.
This causes qmtest-testsuite to be much bigger than testsuite.  I've got a
7MB testsuite directory, and a 54MB qm-testsuite directory, and this is just
for g++.  This could cause disk space problems if you have a lot of toolchains
with a lot of multilibs and you are running entire toolchain tests (e.g.
binutils and gcc and gdb) and they are all leaving files behind.

I get 20 XPASS for qmtest, but not for dejagnu.  This seems to be a problem
with the testsuite notations being ambiguous and/or poorly defined.  Or maybe
it is a problem with dejagnu.

When running the testsuite inside the gui, the intermediate results sometimes
had inconsistent numbers.  For instance, an early update told me that I had
179 PASSes, representing 101% of the 178 total tests that had been run so far.
As the testing progressed, the inconsistencies got bigger.  At one point,
I had a FAIL+UNTESTED+PASS sum which was 4 larger than Total.  I can understand
inconsistencies in the percentages which are rounded, but an inconsistency
in the total number of testcases is surprising.

When I tried to save results inside the gui, netscape chugged for a few minutes
without apparently doing anything.  For a while there I thought it was broken,
because it wasn't doing anything visible.  But then it finally asked me to
approve the filename, and it was done.  This could be a little friendlier.

When I first tried to load results inside the gui, I didn't know what kind of
file it wanted, so I tried browse, and it looked for .html files.  This was
very confusing at the time, since there are no html result files.  It wasn't
until after I saved for the first time that I noticed that a result file really
was a .qmr file, and then I was able to load results successfully.

When I exit the qmtest gui, the "make qmtest-gui" command doesn't exit too.
I have to kill it by hand.

I couldn't help but notice that QMTest documentation has an advertisement for
CodeSourcery services.  Dejagnu doesn't.

I tried looking at the QMTest sources.  They looked a lot cleaner than
dejagnu, but QMTest isn't as mature as dejagnu yet.  Hopefully QMTest will
remain clean.  One thing I didn't see was support for timeouts.  What happens
if the compiler or a testcase gets into an infinite loop?  You have to have
timeouts so that the testsuite will finish.  Also, we need timeouts that are
adjustable, for dealing with slow targets, or slow compilers.

Overall, it was pretty easy to get qmtest working, and it does have some
useful features that dejagnu doesn't have.  It looks hopeful as a possible
dejagnu replacement, but it won't be really useful to me though until it has
support for talking to remote targets, including embedded target boards.

Jim