Re: QMTest and the G++ testsuite

[Alexandre Oliva <aoliva at redhat dot com>]

On May 20, 2002, Mark Mitchell <mark@codesourcery.com> wrote:

> --On Monday, May 20, 2002 01:49:06 PM -0300 Alexandre Oliva
> <aoliva@redhat.com> wrote:

>> On May 20, 2002, Mark Mitchell <mark@codesourcery.com> wrote:

>>> Now, there is the question of writing the interpreters for various
>>> setups; you need the logic to zap the code to the target board, run it,
>>> extract the output, zap it back.

>> How about running the *build* on a separate machine?

> There's no major conceptual problem here; just a matter of
> plugging in the right bits.

And how about say the GDB testsuite, in which it may be necessary to
run GDB on one machine, compile one or more source files on another,
connect to a board and upload the program to it using rcp, NFS, tftp,
kermit or other board-specific mechanisms, and then issue commands to
gdb and verify that they had the correct effect on the program?

This can all be done with DejaGNU, and it's all in place.  If your
plan is to push for DejaGNU to be replaced with QMTest, which is
something I'd welcome, it would be at least nice if it already offered
the features that we currently have in DejaGNU, otherwise there's
little point in switching, since I haven't seen practical advantages
of using QMTest so far (other than the fact that it's written in
Python, which I find infinitely more palatable than tcl/expect, which
is a welcome change for me).

> The important question is whether we think it's worth doing.

I would surely like to move to something that is better documented,
cleaner and easier to use than DejaGNU, but I can't justify simply
dropping on the floor all the work that we've put into DejaGNU to
adopt a different tool just because it has the *potential* of
providing all of the features.  If it is to replace DejaGNU, it had
better have a significantly larger subset of DejaGNU's feature for a
transition to be financially viable, and it would have to offer very
clear advantages over DejaGNU.  As I wrote before, the implementation
language is the only advantage I've noticed so far, but others
disagree with my personal language preferences.  One of the major
selling points of QMTest that I've seen is the ability to easily
compare test results with those of a previous run, but, hey!, we've
had diff for decades now :-)

> your goal should be to answer the question

>   If Mark is right, and this will work in cross environments, would I
>   want to have it?  Or would I rather stick with DejaGNU?

I hope I have kind of answered the question above, but to sum it up: I
like the idea of moving to something that is implemented in Python and
has better documentation, but I don't like the fact that a lot of work
that has already been done in DejaGNU, and that is therefore known to
be possible in DejaGNU, would have to be duplicated to get a new tool
in the same point as the one we use today.  Until this new tool proves
to be as capable of testing in the many different, complex scenarios
in which the current tool is used, I think I'd rather keep on using
the already-proven tool.  Even if a new tool were as capable as what
we have now, I'd probably still prefer to stick to the old tool, just
because there's a migration cost that has to be offset by clear
advantages in the switch.  In the absence of such a clear advantage,
which I'm not sure a change in the implementation language is (as
much as I'd like it to be :-), switching to a new tool is a very hard
proposition.

That said, I'd like to applaud the initiative of getting a new
Free-Software testing tool developed; it's nice to see something new
in this field, and I sure hope it grows to the point of quickly being
a viable alternative for the complex test scenarios required by the
entire toolchain.  It clearly is already capable of the simplest ones
(which happen to be the most commonly used ones too); I hope it
becomes capable of handling the most complex ones in the
not-too-distant future.


Note that this is just my personal opinion; I don't speak for Red Hat,
and I don't want to give the impression that I do, especially in this
posting.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer