This is the mail archive of the gcc@gcc.gnu.org mailing list for the GCC project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: QMTest and the G++ testsuite


On May 22, 2002, Mark Mitchell <mark@codesourcery.com> wrote:

> My goal is to get the process started -- I've offerred to do what it
> takes for GCC.

My concern is that, by going down that path, we may soon get into a
situation in which most people would trade DejaGNU for QMTest because
of the features it offers (and you're doing a good job at showing me
and others that it does have some significant advantages), in such a
way that we may soon start getting chunks of the testsuite that depend
exclusively on QMTest, and then there will be no way back.  At that
point, those who rely on the more complex features of DejaGNU (like
Red Hat), that are of little use for a large number of the users, will
find themselves in a situation of either having to disregard part of
the testsuite, or forced to put resources into getting QMTest on par
with DejaGNU.

I can't say I find either situation unfair, and I wouldn't like to be
the one to prevent others from benefiting from the advantages of
QMTest, but I'd like some assurance that no tests that depend
exclusively on QMTest will be introduced, at least until QMTest can
actually replace DejaGNU without major hassles in the more complex
testing scenarios that Red Hat and probably a couple of other
toolchain-for-embedded-development building and testing businesses
depend on.

> If nobody's interested enough to move on to GDB, well, that will be
> that.

Which will actually increase the time investment for those of us who
test the entire toolchain.  Then, instead of needing a single tool,
namely DejaGNU, one will need two.

> You're right to focus on the costs and benefits.  I just think you've
> missed some of the benefits. :-)

Indeed.  But if it can't be used in the most complex scenarios,
without a lot of additional development effort, it's of no use.  It
doesn't matter than it can run 20 times faster by using 20 host
machines if the bottleneck is the 9600bps serial line that connects to
the single board available for testing.  And if it can't do
build/host/board testing, it won't be just 20 times faster, it will be
infinitely faster, because it won't be usable at all.  Of course, in
this case, being able to compare with earlier test results would be of
no gain, since there won't be any test results, ever :-)

And then, there are many other issues when testing boards; many of the
experimental models we have to work with are unreliable and require
constant resetting, re-uploading and re-running of tests in order to
get reliable results.  DejaGNU already has all of the bits necessary
to do all of this automatically.  A replacement tool would be
extremely painful to use if it couldn't do things like this.

> Note that my first request was to *permit* people to do their testing
> with QMTest.  That allows us to compare the tools side by side --
> without forcing anyone to switch or committing us to switching.

If it were not for my concerns exposed above, of introducing an option
that soon becomes a dependency, I'd be all for it.  Perhaps I'm being
too negative, or too conservative, but I've been burned by such soft
introduction of dependencies too many times to refrain from bringing
these points up in public.

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                  aoliva@{cygnus.com, redhat.com}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]