Re: QMTest and the G++ testsuite

[mike stump <mrs at windriver dot com>]

> Date: Wed, 22 May 2002 23:10:56 -0700
> From: Zack Weinberg <zack@codesourcery.com>

> However, I'd like to point out that there are some categories of
> test that would be difficult or impossible to express within
> DejaGNU, but can be done pretty easily in QMTest.

> The example that comes to mind is the "nasty file names" test, which
> generates a series of input files whose names (collectively) cover
> all the characters that the operating system allows in file names,
> and makes sure that the compiler can handle them.  You can't check
> all those files into CVS, the set of acceptable names varies with
> the operating system.  I tried and failed to do this in DejaGNU.
> Granted, this test requires writing custom Tcl, and would require
> writing custom Python in QMTest.

dejagnu runs tcl to do a test.  tcl is Turing complete.  tcl can
easily do Turing complete things (not as hard as vi say).  It is
trivial to do this sort of test in the exiting framework for either
the C or C++ testsuite, one merely needs to `code' it.

dejagnu could be made to test in parallel.  dejagnu could be crippled
to only report one result per file.

dejagnu allows the trivial management of complex state during a
testsuite run.  QMTest would seem to require one to flush this out to
the filesystem.

dejagnu can do canadian cross testing while managing disparate
filesystems (unshared between machines) trivially.  dejagnu can even
move files using tip, if it has to.

After QMTest has been used in a production environment with diverse
needs and enhanced to support those needs, it will be close to
matching dejagnu.

I don't see that it is advantageous enough to enhance QMTest, rather
than just enhance dejagnu to provide the one or two features your
interested in.

I think we gain more by having a shared standardized test frameworks
for all the GNU tools, than for one tool to be `special'.  That said,
I like the notion that the testsuite contains testcases that and ebb
and flow between vastly different test-harnesses, that _is_ why you
found the g++ testsuite trivial to get working.  I think it would be
good to _canonicalize_ the gcc testsuite so that it can more trivially
be made to work with QMTest.

That said, I don't see a problem checking in additional support for
alternate frameworks, I just don't want to see existing support
toasted.