This is the mail archive of the gcc@gcc.gnu.org mailing list for the GCC project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: QMTest and the G++ testsuite




--On Thursday, May 23, 2002 09:18:55 PM -0400 Jim Wilson 
<wilson@redhat.com> wrote:

> Here are a few issues that I haven't seen mentioned by anyone else yet.

Thanks for taking the time to investigate.

> There is a POSIX standard for test frameworks: POSIX 1003.3.  DejaGNU
> conforms to this standard.  There is some info about this in the DejaGNU
> documentation. This standard was mainly created for testsuites for
> testing POSIX conformance, but it is still a useful standard for generic
> test frameworks.

QMTest almost conforms in the respect with which I am most aware: the
output formats.  QTMest uses PASS, FAIL, UNTESTED, and ERROR.  (POSIX
has UNRESOLVED.)  We should probably add UNRESOLVED.

ERROR reflects an internal error in the testing tool -- either QMTest, or
the extension bits that are running your particular test case.  It always
relects a bug somewhere; I suppose we could transform it into FAIL for
POSIX conformance.

Like DejaGNU we use "XPASS" and "XFAIL".

> DejaGNU support for multiple tests per file is useful for supporting
> non-GNU testsuites.  PlumHall for instance has files that contain
> thousands of tests. It is useful to get an exact count of passes and
> failures from each file. It is unlikely that we will convince PlumHall
> that their testsuite was designed wrong, and it would be inconvenient to
> try to write a test framework that explicitly reports each individual
> test.

This reflects a misunderstanding, I think.

QMTest is not "one test per file".  It is "one result per test", just
like DejaGNU.

In the G++ testsuite port, I chose to implement one result per file,
because I considered each file a single test.  I could have considered
each file to have N tests, as DejaGNU does -- and it wouldn't be
particularly hard to do that.  I just didn't think it was worth it.

I've used Plum Hall before, and you could do the same thing there.

The other alternative is to use QMTest's "annotations".  QMTest can
add structured information to a test result.  You could produce a single
result but add information indicating how many passes and how many
failures there are.  This output is displayed in the "full" output
format.

> QMTest includes code from third parties with varying copyright notices.

Our lawyers believe we're in compliance with the various licenses, but
we're also going to dramatically simplify this situation in the next
release.  There is no longer any need for us to have the xmlrpc, PyXML,
sgmlop, or gadfly packages bundled in.

We will be keeping only the zope-dtml package.

> I haven't tried running it, but I will try to find time for that so I can
> give more comments.

Thank you.

(Loren has reported a parallelization bug -- ack! -- and I haven't had
time to investigate further yet; if you're interested in that, you may
wish to wait until we get it fixed.)

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]