This is the mail archive of the
mailing list for the GCC project.
QMTest and the G++ testsuite
- From: Mark Mitchell <mark at codesourcery dot com>
- To: gcc at gcc dot gnu dot org
- Date: Mon, 20 May 2002 00:18:18 -0700
- Subject: QMTest and the G++ testsuite
I am about to check in a patch on the mainline to allow the use of
QMTest -- a free testing tool developed by CodeSourcery -- instead of
DejaGNU, to run the G++ regression test suite. (For the patch, see
My objectives are:
1. To convert the entire GCC testsuite to QMTest.
2. To obtain consensus in the community that we should use QMTest as
our testing tool, rather than DejaGNU.
We believe there are several advantages to QMTest, vis a vis DejaGNU,
1. Support for parallel testing; multiple test cases can be run at
once, thereby speeding up testing on multiprocessor machines (and
even uniprocessor machines, since our testing is I/O bound).
2. Support for direct comparisons between test results. QMTest can
easily answer the question "Did I break anything relative to the
results I have from yesterday?"
I realize that realizing the objectives above will take a fair amount
of work, and that it may not succeed at all. If QMTest isn't able to
do a better job than DejaGNU, it certainly will not.
QMTest is a young tool, and there are doubtless going to be issues:
QMTest may not work on some platforms, or there may be bugs that
affect usability, or ...
Let's try to keep the discussion focused on how to get results with
QMTest that are at least as good as we get with DejaGNU. If we use
this as a forum for talking about everyone's personal viewpoint on
testing on general and "I've always really wanted to see X ability",
we're going to get too distracted.
I will be willing to do the work required to get something that people
are happy with -- both in QMTest and in the extension classes that
handle the GCC testsuite -- within reason. I would appreciate help in
trying out QMTest and delineating what, if any, improvements are
needed before we can move to QMTest instead of DejaGNU. I would also
appreciate people making as few changes as possible to the DejaGNU
harness since I'll be having to keep up on the QMTest side. (I'll
probably make some changes to the testsuite as time goes by to try to
make things more uniform; that will be a win no matter what tool we
use. There's really no reason for us to have so many different ways
of doing things in the testsuite.)
I think the first step is to try get a consensus that it is OK to do
your G++ regression-testing with QMTest, instead of DejaGNU, before
check-in. I've verified that I get exactly the same results with
DejaGNU and with QMTest. If we can agree that it's OK to use either
tool, that will give us a transition period where we can iron out the
issues that remain.
I've attached the file README.QMTEST, now present in the "testsuite"
directory, that explains how to try out "make qmtest-g++" instead of
Mark Mitchell firstname.lastname@example.org
CodeSourcery, LLC http://www.codesourcery.com
Testing with QMTest
You can use QMTest to test G++. (In the future, it may be possible to
test other parts of GCC with QMTest as well, but it is not possible
The use of QMTest to run the G++ tests has not been approved as an
officially supported testing procedure. Therefore, you must run the
tests using DejaGNU (with "make check-g++") before committing changes
that affect G++.
Differences from DejaGNU
Although QMTest can be used to run the G++ test suite, it works
somewhat differently from DejaGNU. In particular:
- In DejaGNU, a single source file contains many tests. Each
line where a diagnostic is expected is considered a separate
test. Testing for successful compilation and testing for
successful execution of the generated program are considered
separate tests. Thus, a test "test.C" could contain, say,
seven tests; some of which might pass and some of which might
With QMTest, each source file is considered a single test. If any
of the seven sub-tests fail, the entire test is considered to fail.
However, QMTest does present information about *why* the test
failed, so the same information is effectively available.
It is true that, therefore, causing an already failing test to "fail
more" is not immediately detectable through an additional unexpected
failure messages when using QMTest. On the other hand, most people
seem to think of each source file as "a test", not "twelve tests",
so the model QMTest uses may be more natural.
- In DejaGNU, tests themselves keep track of expected and unexpected
failures. The QMTest philosophy is that expected failures should be
stored separately from the tests themselves; in particular, that
tonights results can be tomorrow's expectations. In order to
preserve compatibility with DejaGNU, the first time you use QMTest
to test G++, QMTest computes the set of expected failures indicated
by the tests, and then compares the actual results with these
results. Therefore, if you change the expected failure notations in
the DejaGNU tests, you must rebuild the set of expected failures.
To do this, remove the file "qmtestsuite/gpp-expected.qmr". Then,
when you rerun the tests, the expected failures will be
You must download and install the following software:
- Python 2.1 (or greater)
You may already have Python on your system; in particular, many
GNU/Linux systems ship with Python installed.
Installation instructions are available on the web-site.
- QMTest 1.1.4 (or greater)
QMTest is available at:
Installation instructions are available on the web-site.
- QMTC 1.1
This package is available from:
See the file called INSTALL in the distribution.
Running the Tests
To run the tests, run "make qmtest-g++" in the gcc directory of your
build tree. The first time that you do this, QMTest will calculate
the set of tests that are expected to fail on your platform, so it
will take several minutes before you see any test results. After the
first time, QMTest will start running the tests much more quickly.
If the test summary printed at the test run indicates no unexpected
failures, then G++ is behaving as expected on your target. (Some
unexpected passes are normal.)
You can obtain detailed information about why tests failed in one
of two ways:
1. By invoking QMTest with the "-f full" option. For example:
make QMTESTRUNFLAGS="-f full" qmtest-g++
2. Examining the log file qmtestsuite/gpp.qmr after the tests have
Here are some more advanced usage instructions:
1. To run a particular set of tests (rather than all of the tests),
use the make variable "QMTEST_GPP_TESTS". For example,
make QMTEST_GPP_TESTS="gpp.dg" qmtest-g++
will run only the tests in the g++.dg subdirectory, and:
make QMTEST_GPP_TESTS="gpp.dg.special.conpr1 \
will run only the two tests indicated.
2. To run qmtest with particular flags, use the make variables
"QMTESTFLAGS" and "QMTESTRUNFLAGS". For example:
make QMTESTFLAGS="-v" QMTESTRUNFLAGS="-f full" qmtest-g++
will run qmtest like this:
qmtest -v run -f full ...
3. To run the compiler with particular flags, use QMTESTRUNFLAGS to
set the QMTest context variable "GPPTest.flags", like this:
make QMTESTRUNFLAGS='-c GPPTest.flags="-funroll-loops"' qmtest-g++
The compiler will then use the "-funroll-loops" switch when
4. If qmtest is not in your path, you can indicate the full path to
QMTest by using the make variable "QMTEST_PATH", like this:
make QMTEST_PATH=/path/to/qmtest qmtest-g++
5. To start the QMTest GUI, use:
(Note that this will run the program called "netscape" in your path.
If you want to use another browser, you must configure qmtest as
described in its manual.)
Bear in mind that the QMTest GUI is insecure; malicious users with
access to your machine may be able to run commands as if they were
you. The QMTest GUI only binds to the loopback IP addresss, which
provides a measure of security, but not enough for use in untrusted
6. If you have a multiprocessor, you can run the tests in parallel by
passing the "-j" option to qmtest:
make QMTESTRUNFLAGS="-j 4" qmtest-g++
will run tests in four threads. (It is also possible to run tests
across multiple machines; for more information see the QMTest
7. If you are testing a cross compiler, you must specify an interpreter
that is capable of running the generated program. It must be a
program "p" such that:
p program arg1 arg2 arg3 ...
behaves exactly like running:
program arg1 arg2 arg3 ...
would on the target machine. You specify this program via the
"CompilerTest.interpreter" context variable:
make QMTESTRUNFLAGS='-c CompilerTest.interpreter=/path/to/interpreter'