QMTest Update

[Mark Mitchell]

--On Wednesday, June 12, 2002 07:06:40 PM -0400 Jim Wilson 
<wilson@redhat.com> wrote:

> I finally got around to trying QMTest.

Thank you for doing that and for the very detailed feedback!

I very much appreciate the comments and criticisms.

> The docs say that Python 2.1 (or greater) is required.  So I tried the
> version the docs recommended (2.1.3) and discovered that it doesn't work.
> QMTest is using a python function (PyType_IsSubtype) that only exists in
> Python 2.2. The docs in qmtest and gcc need to be fixed.

Curious; I'm using Python 2.1.1 on one machine without a problem.  I will
poke around and see if I can figure out what is what.

> Testcase results get reported in a non-deterministic order.

Yes; I still believe that running in a random order is a useful
feature, but it will no longer be the default with QMTest 2.0.  That
was a mistake in QMTest 1.x.

> QMTest is leaving behind the final file for many testcases.

Good point.

> I get 20 XPASS for qmtest, but not for dejagnu.  This seems to be a
> problem with the testsuite notations being ambiguous and/or poorly
> defined.  Or maybe it is a problem with dejagnu.

These seem to come from "// Crash test - XFAIL *-*-*" where the compiler
does not crash.  I believe those should indeed be XPASS, but perhaps I've
misinterpreted the DejaGNU results.

> When running the testsuite inside the gui, the intermediate results
> sometimes had inconsistent numbers.

Thank you; we'll figure this out.

> When I tried to save results inside the gui, netscape chugged for a few
> minutes without apparently doing anything.

Indeed; we need some kind of progress meter.

> files.  It wasn't until after I saved for the first time that I noticed
> that a result file really was a .qmr file, and then I was able to load
> results successfully.

Some browsers will automatically look for .qmr files; others ignore the
appropriate JavaScript.  We will make the little pop-up box clearer.

-- 
Mark Mitchell                mark@codesourcery.com
CodeSourcery, LLC            http://www.codesourcery.com