This is the mail archive of the
gcc-patches@gcc.gnu.org
mailing list for the GCC project.
PATCH: Update QMTest info in G++
- From: Mark Mitchell <mark at codesourcery dot com>
- To: gcc-patches at gcc dot gnu dot org
- Cc: phil at jaj dot com
- Date: Wed, 4 Jun 2003 17:20:15 -0700
- Subject: PATCH: Update QMTest info in G++
- Reply-to: mark at codesourcery dot com
At the GCC Summit, several people asked me how we used QMTest for
testing G++ these days.
This patch updates the instructions and Makefile targets for current
versions of QMTest (which can now emulate DejaGNU even more closely).
We're working on running the GCC (as well as G++) testsuite, and
making good progress on that front as well.
Tested by using the QMTest Makefile targets; applied on the mainline.
--
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com
2003-06-04 Mark Mitchell <mark@codesourcery.com>
* Makefile.in (QMTESTRUNFLAGS): Set for DejaGNU emulation.
(QMTEST_GPP_TESTS): Use "g++" by default.
(stamp-qmtest): Tweak database creation.
(QMTEST_DIR/context): Update context file format.
(qmtest-g++): Tweak command-line.
2003-06-04 Mark Mitchell <mark@codesourcery.com>
* README.QMTEST: Update.
Index: Makefile.in
===================================================================
RCS file: /cvs/gcc/gcc/gcc/Makefile.in,v
retrieving revision 1.1064
diff -c -5 -p -r1.1064 Makefile.in
*** Makefile.in 4 Jun 2003 07:51:39 -0000 1.1064
--- Makefile.in 5 Jun 2003 00:10:50 -0000
*************** QMTEST_PATH=qmtest
*** 3376,3441 ****
# The flags to pass to qmtest.
QMTESTFLAGS=
# The flags to pass to "qmtest run".
! QMTESTRUNFLAGS=
# The command to use to invoke qmtest.
QMTEST=${QMTEST_PATH} ${QMTESTFLAGS}
# The tests (or suites) to run.
! QMTEST_GPP_TESTS=gpp
# The subdirectory of the OBJDIR that will be used to store the QMTest
# test database configuration and that will be used for temporary
# scratch space during QMTest's execution.
QMTEST_DIR=qmtestsuite
# Create the QMTest database configuration.
${QMTEST_DIR} stamp-qmtest:
! debug_options=""; \
! ${STAMP} empty.C; \
! for option in \
! -gdwarf-2 -gstabs -gstabs+ -gxcoff -gxcoff+ -gcoff; do \
! (./cc1plus -q $${option} empty.C 2>&1 | \
! grep "unknown or unsupported -g option" > /dev/null) || \
! debug_options="$${debug_options}$${option} "; done; \
! ${QMTEST} -D ${QMTEST_DIR} create-tdb \
! -c gcc_database.GCCDatabase \
! -a GCCDatabase.testsuite_root=`cd ${srcdir}/testsuite && pwd` \
! -a GCCDatabase.debug_options="$${debug_options}"
! rm -f empty.C empty.s
! $(STAMP) stamp-qmtest
# Create the QMTest context file.
${QMTEST_DIR}/context: stamp-qmtest
! echo "GCCTest.flags=-B${objdir}" >> $@
! echo "GCCTest.objdir=${objdir}/.." >> $@
! echo "GCCTest.host=${host_canonical}" >> $@
! echo "GCCTest.target=${target}" >> $@
! echo "GCCTest.gcov=${objdir}/gcov" >> $@
! echo "GPPTest.gpp=${objdir}/g++" >> $@
! echo "DGTest.demangler=${objdir}/c++filt" >> $@
# Run the G++ testsuite using QMTest.
! qmtest-g++: ${QMTEST_DIR}/context ${QMTEST_DIR}/gpp-expected.qmr
cd ${QMTEST_DIR} && ${QMTEST} run ${QMTESTRUNFLAGS} -C context \
! -o gpp.qmr -O gpp-expected.qmr \
! ${QMTEST_GPP_TESTS}
# Use the QMTest GUI.
qmtest-gui: ${QMTEST_DIR}/context
cd ${QMTEST_DIR} && ${QMTEST} gui -C context
-
- # Build the set of expected G++ failures.
- ${QMTEST_DIR}/gpp-expected.qmr: ${QMTEST_DIR}/context
- echo "Determining expected results..."
- cd ${QMTEST_DIR} && ${QMTEST} run ${QMTESTRUNFLAGS} -C context \
- -c "GCCTest.generate_xfails=1" -o gpp-expected.qmr \
- ${QMTEST_GPP_TESTS} \
- > /dev/null
.PHONY: qmtest-g++
# Run Paranoia on real.c.
--- 3376,3425 ----
# The flags to pass to qmtest.
QMTESTFLAGS=
# The flags to pass to "qmtest run".
! QMTESTRUNFLAGS=-f none --result-stream dejagnu_stream.DejaGNUStream
# The command to use to invoke qmtest.
QMTEST=${QMTEST_PATH} ${QMTESTFLAGS}
# The tests (or suites) to run.
! QMTEST_GPP_TESTS=g++
# The subdirectory of the OBJDIR that will be used to store the QMTest
# test database configuration and that will be used for temporary
# scratch space during QMTest's execution.
QMTEST_DIR=qmtestsuite
# Create the QMTest database configuration.
${QMTEST_DIR} stamp-qmtest:
! ${QMTEST} -D ${QMTEST_DIR} create-tdb \
! -c gcc_database.GCCDatabase \
! -a srcdir=`cd ${srcdir}/testsuite && pwd` && \
! $(STAMP) stamp-qmtest
# Create the QMTest context file.
${QMTEST_DIR}/context: stamp-qmtest
! rm -f $@
! echo "CompilerTable.languages=c cplusplus" >> $@
! echo "CompilerTable.c_kind=GCC" >> $@
! echo "CompilerTable.c_path=${objdir}/xgcc" >> $@
! echo "CompilerTable.c_options=-B${objdir}/" >> $@
! echo "CompilerTable.cplusplus_kind=GCC" >> $@
! echo "CompilerTable.cplusplus_path=${objdir}/g++" >> $@
! echo "CompilerTable.cplusplus_options=-B${objdir}/" >> $@
! echo "DejaGNUTest.target=$(target)" >> $@
# Run the G++ testsuite using QMTest.
! qmtest-g++: ${QMTEST_DIR}/context
cd ${QMTEST_DIR} && ${QMTEST} run ${QMTESTRUNFLAGS} -C context \
! -o g++.qmr ${QMTEST_GPP_TESTS}
# Use the QMTest GUI.
qmtest-gui: ${QMTEST_DIR}/context
cd ${QMTEST_DIR} && ${QMTEST} gui -C context
.PHONY: qmtest-g++
# Run Paranoia on real.c.
Index: testsuite/README.QMTEST
===================================================================
RCS file: /cvs/gcc/gcc/gcc/testsuite/README.QMTEST,v
retrieving revision 1.3
diff -c -5 -p -r1.3 README.QMTEST
*** testsuite/README.QMTEST 26 Sep 2002 19:49:48 -0000 1.3
--- testsuite/README.QMTEST 5 Jun 2003 00:10:51 -0000
*************** yet.)
*** 8,125 ****
The use of QMTest to run the G++ tests has not been approved as an
officially supported testing procedure. Therefore, you must run the
tests using DejaGNU (with "make check-g++") before committing changes
that affect G++.
! Differences from DejaGNU
! ========================
! Although QMTest can be used to run the G++ test suite, it works
! somewhat differently from DejaGNU. In particular:
! - In DejaGNU, a single source file contains many tests. Each
! line where a diagnostic is expected is considered a separate
! test. Testing for successful compilation and testing for
! successful execution of the generated program are considered
! separate tests. Thus, a test "test.C" could contain, say,
! seven tests; some of which might pass and some of which might
! fail.
!
! With QMTest, each source file is considered a single test. If any
! of the seven sub-tests fail, the entire test is considered to fail.
! However, QMTest does present information about *why* the test
! failed, so the same information is effectively available.
!
! It is true that, therefore, causing an already failing test to "fail
! more" is not immediately detectable through an additional unexpected
! failure messages when using QMTest. On the other hand, most people
! seem to think of each source file as "a test", not "twelve tests",
! so the model QMTest uses may be more natural.
!
! - In DejaGNU, tests themselves keep track of expected and unexpected
! failures. The QMTest philosophy is that expected failures should be
! stored separately from the tests themselves; in particular, that
! tonights results can be tomorrow's expectations. In order to
! preserve compatibility with DejaGNU, the first time you use QMTest
! to test G++, QMTest computes the set of expected failures indicated
! by the tests, and then compares the actual results with these
! results. Therefore, if you change the expected failure notations in
! the DejaGNU tests, you must rebuild the set of expected failures.
!
! To do this, remove the file "qmtestsuite/gpp-expected.qmr". Then,
! when you rerun the tests, the expected failures will be
! automatically recalculated.
Setting Up
==========
You must download and install the following software:
! - Python 2.1 (or greater)
See http://www.python.org.
You may already have Python on your system; in particular, many
GNU/Linux systems ship with Python installed.
Installation instructions are available on the web-site.
! - QMTest 1.1.4 (or greater)
! See http://www.qmtest.com.
! QMTest is available at:
! http://www.codesourcery.com/qm/qmtest_download
!
! or:
!
! ftp://ftp.codesourcery.com/pub/qmtest
!
! Installation instructions are available on the web-site.
!
! - QMTC 1.1
! This package is available from:
! ftp://ftp.codesourcery.com/pub/qmtest/qmtc/qmtc-<version>.tar.gz
!
! See the file called INSTALL in the distribution.
Running the Tests
=================
! To run the tests, run "make qmtest-g++" in the gcc directory of your
! build tree. The first time that you do this, QMTest will calculate
! the set of tests that are expected to fail on your platform, so it
! will take several minutes before you see any test results. After the
! first time, QMTest will start running the tests much more quickly.
!
! If the test summary printed at the test run indicates no unexpected
! failures, then G++ is behaving as expected on your target. (Some
! unexpected passes are normal.)
!
! You can obtain detailed information about why tests failed in one
! of two ways:
! 1. By invoking QMTest with the "-f full" option. For example:
!
! make QMTESTRUNFLAGS="-f full" qmtest-g++
!
! 2. Examining the log file qmtestsuite/gpp.qmr after the tests have
! run.
Here are some more advanced usage instructions:
1. To run a particular set of tests (rather than all of the tests),
use the make variable "QMTEST_GPP_TESTS". For example,
make QMTEST_GPP_TESTS="g++.dg" qmtest-g++
will run only the tests in the g++.dg subdirectory, and:
! make QMTEST_GPP_TESTS="g++.dg/special/conpr1.C \
g++.old-deja/g++.other/access2.C"
qmtest-g++
will run only the two tests indicated.
--- 8,109 ----
The use of QMTest to run the G++ tests has not been approved as an
officially supported testing procedure. Therefore, you must run the
tests using DejaGNU (with "make check-g++") before committing changes
that affect G++.
! QMTest emulates DejaGNU behavior very closely when running the tests.
! QMTest has two output modes: a DejaGNU emulation mode and a native
! QMTest mode.
! In the DejaGNU mode, you should receive output that is almost exactly
! the same as the DejaGNU output; in particular, you should see the same
! number of passes, failures, etc. When using the DejaGNU-style output,
! QMTest uses the "xfail" indications in the test cases to determine
! which tests are expected to pass and which are expected to fail, and
! presents that information in the same way as DejaGNU.
!
! In the QMTest mode, the number of passes and failures will be
! different from that obtained when using DejaGNU. The reason is that a
! single source file may contain multiple DejaGNU tests. In DejaGNU,
! each line where a diagnostic is expected is considered a separate
! test. Testing for successful compilation and testing for successful
! execution of the generated program are considered separate tests. So,
! a single source file "test.C" could contain, say, seven tests; some of
! which might pass and some of which might fail.
!
! In the QMTest mode, each source file is considered a single test. If
! any of the seven sub-tests fail, the entire test is considered to
! fail. However, QMTest does present information about *why* the test
! failed, so the same information is effectively available.
!
! In the QMTest mode, whether or not a test is expected to fail is
! determined not by an indication in the test, but rather by comparing
! the new results to the results of a previous run. Testing for whether
! a change caused a regression is very simple: run the tests before
! making the change, run them again after making the change, and let
! QMTest compare the results.
!
! The mode chosen only affects the output from QMTest, not how it runs
! the tests or how it stores the data. Therefore, if you choose to run
! in the QMTest mode and later want to get the DejaGNU style output, or
! vice versa, you can do that as described below.
Setting Up
==========
You must download and install the following software:
! - Python 2.2 (or greater)
See http://www.python.org.
You may already have Python on your system; in particular, many
GNU/Linux systems ship with Python installed.
Installation instructions are available on the web-site.
! - A current version of QMTest. No released version provides all of
! the functionality required, so you must obtain QMTest from CVS.
! To do that, follow the instructions at:
! http://www.codesourcery.com/qm/qmtest_development
! Installation instructions are available in the file called README
! after you check out QMTest.
! - The "qmtc" and "qmtest_gcc" QMTest support packages. These are
! available from the same CVS repository as QMTest. For example, to
! check out "qmtc", do:
! cvs -d :pserver:anoncvs@cvs.codesourcery.com:/home/qm/Repository \
! co qmtc
+ You do not have to install these packages; you need only check them
+ out.
+
Running the Tests
=================
! First, you must set QMTEST_CLASS_PATH so that it can find the qmtc and
! qmtest_gcc support packages:
!
! export QMTEST_CLASS_PATH=/path/to/qmtc:/path/to/qmtest_gcc
! The, run "make qmtest-g++" in the gcc directory of your build tree.
Here are some more advanced usage instructions:
1. To run a particular set of tests (rather than all of the tests),
use the make variable "QMTEST_GPP_TESTS". For example,
make QMTEST_GPP_TESTS="g++.dg" qmtest-g++
will run only the tests in the g++.dg subdirectory, and:
! make QMTEST_GPP_TESTS="g++.dg/special/conpr-1.C \
g++.old-deja/g++.other/access2.C"
qmtest-g++
will run only the two tests indicated.
*************** Here are some more advanced usage instru
*** 130,143 ****
will run qmtest like this:
qmtest -v run -f full ...
3. To run the compiler with particular flags, use QMTESTRUNFLAGS to
! set the QMTest context variable "GPPTest.flags", like this:
! make QMTESTRUNFLAGS='-c GPPTest.flags="-funroll-loops"' qmtest-g++
The compiler will then use the "-funroll-loops" switch when
compiling.
4. If qmtest is not in your path, you can indicate the full path to
--- 114,133 ----
will run qmtest like this:
qmtest -v run -f full ...
+ (The "-f full" mode will provide detailed information about each
+ test as it runs.)
+
3. To run the compiler with particular flags, use QMTESTRUNFLAGS to
! set the QMTest context variable "CompilerTable.cplusplus_options",
! like this:
! make \
! QMTESTRUNFLAGS='-c CompilerTable.cplusplus_options="-funroll-loops"' \
! qmtest-g++
The compiler will then use the "-funroll-loops" switch when
compiling.
4. If qmtest is not in your path, you can indicate the full path to
*************** Here are some more advanced usage instru
*** 147,157 ****
5. To start the QMTest GUI, use:
make qmtest-gui
! (Note that this will run the program called "netscape" in your path.
If you want to use another browser, you must configure qmtest as
described in its manual.)
Bear in mind that the QMTest GUI is insecure; malicious users with
access to your machine may be able to run commands as if they were
--- 137,147 ----
5. To start the QMTest GUI, use:
make qmtest-gui
! (Note that this will run the program called "mozilla" in your path.
If you want to use another browser, you must configure qmtest as
described in its manual.)
Bear in mind that the QMTest GUI is insecure; malicious users with
access to your machine may be able to run commands as if they were
*************** Here are some more advanced usage instru
*** 166,187 ****
will run tests in four threads. (It is also possible to run tests
across multiple machines; for more information see the QMTest
manual.)
! 7. If you are testing a cross compiler, you must specify an interpreter
! that is capable of running the generated program. It must be a
! program "p" such that:
!
! p program arg1 arg2 arg3 ...
!
! behaves exactly like running:
!
! program arg1 arg2 arg3 ...
!
! would on the target machine. You specify this program via the
! "CompilerTest.interpreter" context variable:
!
! make QMTESTRUNFLAGS='-c CompilerTest.interpreter=/path/to/interpreter'
! qmtest-g++
--- 156,168 ----
will run tests in four threads. (It is also possible to run tests
across multiple machines; for more information see the QMTest
manual.)
! 7. If a test (say "g++.dg/abi/bitfield1.C") fails, and you want to get
! more detailed information, you can do:
+ cd qmtestsuite
+ qmtest summarize g++.qmr g++.dg/abi/bitfield1.C
+ to get more information about the commands that were run and the
+ output produced.