This is the mail archive of the
mailing list for the GCC project.
Re: QMTest and the G++ testsuite
On Wed, May 22, 2002 at 11:41:52PM -0700, mike stump wrote:
> > The example that comes to mind is the "nasty file names" test, which
> > generates a series of input files whose names (collectively) cover
> > all the characters that the operating system allows in file names,
> > and makes sure that the compiler can handle them. You can't check
> > all those files into CVS, the set of acceptable names varies with
> > the operating system. I tried and failed to do this in DejaGNU.
> > Granted, this test requires writing custom Tcl, and would require
> > writing custom Python in QMTest.
> dejagnu runs tcl to do a test. tcl is Turing complete. tcl can
> easily do Turing complete things (not as hard as vi say). It is
> trivial to do this sort of test in the exiting framework for either
> the C or C++ testsuite, one merely needs to `code' it.
Turing-completeness doesn't mean a hell of a lot. Yes, tcl is easier
to work with than vi, or INTERCAL. It is still a language that a lot
of people (including myself) find nearly impossible to write anything
Also, the language isn't what's at stake here. A test harness is,
fundamentally, a set of library modules for whatever language. A
discussion of the merits of QMTest vs DejaGNU should be a discussion
of those library modules, how easily they allow people to express
common test cases and common testing scenarios, how comprehensive
their built-in features are, and how easy it is to extend them if the
built-in features don't cover the territory.
You claim that this test is expressible in DejaGNU as it stands. I
can only say that I tried and failed. I would be delighted to get a
gcc.misc-tests .exp file that implements this test (it's the only way
we can be sure the linemarker bugs you found the other month, don't
come back). I admit to not having tried to implement this in QMTest -
> dejagnu can do canadian cross testing while managing disparate
> filesystems (unshared between machines) trivially. dejagnu can even
> move files using tip, if it has to.
> After QMTest has been used in a production environment with diverse
> needs and enhanced to support those needs, it will be close to
> matching dejagnu.
On this subject, I am seriously wondering whether the scenarios people
are posting are *really* the scenarios they need to be using, or
whether they are actually working around inadequacies of DejaGNU and
don't even realize it. Take for example the horrific mess that is
your VxWorks test harness. I spent three days, eight hours each day,
attempting to get it to work in any context other than Samuel's magic
script. I failed. And I failed because of genuine, fundamental flaws
in DejaGNU's target-board abstraction. A good two-thirds of
lib/vxworks.exp is kludges around those flaws. Yeah, it's the kludges
that don't work in any context other than Samuel's magic script
running on a Solaris box in the Alameda office of Wind River. But the
kludges have to be there, and they have to have context dependencies!
Meantime, the actual task to be performed - reserve a target board,
bring it up from its serial console, attach a target server and a
windshell process, then run tests by issuing commands to the windshell
- is quite easy to express with QMTest resources. I could knock it
out in maybe four hours tops. And it would have *no* context