Re: QMTest and the G++ testsuite

[Daniel Jacobowitz <dmj+ at andrew dot cmu dot edu>]

On Wed, May 22, 2002 at 04:56:03PM -0700, Mark Mitchell wrote:
> In neither case is their really *built-in* support for much more than
> that.  DejaGNU comes with a useful library full of routines that help
> you do various things, including interactive testing and funky
> cross-machine magic.  As Alexandre has pointed out, QMTest's library
> is smaller at the moment.

I'm not sure you can really dismiss Expect this way; you'd need an
entire Python equivalent to Expect in order to do interactive testing
and such a beast would be quite difficult to write.  I do see some
potential in wrapping this around the existing TCL tests for GDB,
though... and GDB testing is even less processor-bound than GCC testing
so I expect substantial performance advantages if we could actually get
it to work.

> The answer to "can it cope with interactive testing" is "yes".  In the
> reduction-to-absurdity case QMTest test cases can invoke DejaGNU and
> report the results.  (There are actually reasons to do that, but
> we don't need to get into them here.)  Yes, you can use expect.  Or,
> you can do something else -- whatever you like.
> 
> There is, as with any tool, a migration cost.
> 
> Note that the cost, however, is not linear in the number of test cases;
> it's linear in the number of kinds of testing you're doing.  For example,
> converting the G++ testsuite didn't involve touching the tests at all;
> it wasn't really very much work.
> 
> As a measure, it took fewer than 1000 lines of code to support the entire
> G++ testsuite; it took only a couple of afternoons.  Much of that work
> will also be useful in the GCC testsuite, and some in GDB as well.
> (There are bits that know how to compile code, for example, that can be
> sahred.)  Some of that work has also given us ideas about how to provide
> more features in QMTest directly; when we do that, even more of the
> code will disappear.

In interactive program testing, "linear in the number of kinds of
testing you're doing" is harder to separate from "linear in the number
of test cases" than you might like to let on.  I've tried many times to
do the equivalent of the very nice drop-in-a-C-file testing that GCC
has in GDB, and I've never found a meaningful way to do it.  Each new
test requires a substantial amount of TCL.

I'd posit that adding parallel testing to DejaGNU on boards which
support it (a relatively small set of the supported targets; really
only native and rsh/ssh based testing and not any embedded target)
would be substantially easier.  The XML patches I mentioned give room
for most of QMTest's other benefits.

Please don't take this wrong; a clean, modern testing framework is a
wonderful thing.  I'm profoundly glad that you wrote one.  But it has a
long way to go before it could ever catch up to DejaGNU in utility.

-- 
Daniel Jacobowitz                           Debian GNU/Linux Developer
MontaVista Software                               Debian Security Team