gcc/testsuite/README.QMTEST

   1 Testing with QMTest
   2 ===================
   3
   4 You can use QMTest to test G++.  (In the future, it may be possible to
   5 test other parts of GCC with QMTest as well, but it is not possible
   6 yet.)
   7
   8 The use of QMTest to run the G++ tests has not been approved as an
   9 officially supported testing procedure.  Therefore, you must run the
  10 tests using DejaGNU (with "make check-g++") before committing changes
  11 that affect G++.
  12
  13 QMTest emulates DejaGNU behavior very closely when running the tests.
  14
  15 QMTest has two output modes: a DejaGNU emulation mode and a native
  16 QMTest mode.
  17
  18 In the DejaGNU mode, you should receive output that is almost exactly
  19 the same as the DejaGNU output; in particular, you should see the same
  20 number of passes, failures, etc.  When using the DejaGNU-style output,
  21 QMTest uses the "xfail" indications in the test cases to determine
  22 which tests are expected to pass and which are expected to fail, and
  23 presents that information in the same way as DejaGNU.
  24
  25 In the QMTest mode, the number of passes and failures will be
  26 different from that obtained when using DejaGNU.  The reason is that a
  27 single source file may contain multiple DejaGNU tests.  In DejaGNU,
  28 each line where a diagnostic is expected is considered a separate
  29 test.  Testing for successful compilation and testing for successful
  30 execution of the generated program are considered separate tests.  So,
  31 a single source file "test.C" could contain, say, seven tests; some of
  32 which might pass and some of which might fail.
  33
  34 In the QMTest mode, each source file is considered a single test.  If
  35 any of the seven sub-tests fail, the entire test is considered to
  36 fail.  However, QMTest does present information about *why* the test
  37 failed, so the same information is effectively available.
  38
  39 In the QMTest mode, whether or not a test is expected to fail is
  40 determined not by an indication in the test, but rather by comparing
  41 the new results to the results of a previous run.  Testing for whether
  42 a change caused a regression is very simple: run the tests before
  43 making the change, run them again after making the change, and let
  44 QMTest compare the results.
  45
  46 The mode chosen only affects the output from QMTest, not how it runs
  47 the tests or how it stores the data.  Therefore, if you choose to run
  48 in the QMTest mode and later want to get the DejaGNU style output, or
  49 vice versa, you can do that as described below.
  50
  51 Setting Up
  52 ==========
  53
  54 You must download and install the following software:
  55
  56 - Python 2.2 (or greater)
  57
  58   See http://www.python.org.
  59
  60   You may already have Python on your system; in particular, many
  61   GNU/Linux systems ship with Python installed.
  62
  63   Installation instructions are available on the web-site.
  64
  65 - A current version of QMTest.  No released version provides all of
  66   the functionality required, so you must obtain QMTest from CVS.
  67
  68   To do that, follow the instructions at:
  69
  70     http://www.codesourcery.com/qm/qmtest_development
  71
  72   Installation instructions are available in the file called README
  73   after you check out QMTest.
  74
  75 - The "qmtc" and "qmtest_gcc" QMTest support packages.  These are
  76   available from the same CVS repository as QMTest.  For example, to
  77   check out "qmtc", do:
  78
  79     cvs -d :pserver:anoncvs@cvs.codesourcery.com:/home/qm/Repository \
  80       co qmtc
  81
  82   You do not have to install these packages; you need only check them
  83   out.
  84
  85 Running the Tests
  86 =================
  87
  88 First, you must set QMTEST_CLASS_PATH so that it can find the qmtc and
  89 qmtest_gcc support packages:
  90
  91   export QMTEST_CLASS_PATH=/path/to/qmtc:/path/to/qmtest_gcc
  92
  93 The, run "make qmtest-g++" in the gcc directory of your build tree.
  94
  95 Here are some more advanced usage instructions:
  96
  97 1. To run a particular set of tests (rather than all of the tests),
  98    use the make variable "QMTEST_GPP_TESTS".  For example,
  99
 100      make QMTEST_GPP_TESTS="g++.dg" qmtest-g++
 101
 102    will run only the tests in the g++.dg subdirectory, and:
 103
 104      make QMTEST_GPP_TESTS="g++.dg/special/conpr-1.C \
 105                             g++.old-deja/g++.other/access2.C"
 106           qmtest-g++
 107
 108    will run only the two tests indicated.
 109
 110 2. To run qmtest with particular flags, use the make variables
 111    "QMTESTFLAGS" and "QMTESTRUNFLAGS".  For example:
 112
 113       make QMTESTFLAGS="-v" QMTESTRUNFLAGS="-f full" qmtest-g++
 114
 115    will run qmtest like this:
 116
 117       qmtest -v run -f full ...
 118
 119    (The "-f full" mode will provide detailed information about each
 120    test as it runs.)
 121
 122 3. To run the compiler with particular flags, use QMTESTRUNFLAGS to
 123    set the QMTest context variable "CompilerTable.cplusplus_options",
 124    like this:
 125
 126       make \
 127         QMTESTRUNFLAGS='-c CompilerTable.cplusplus_options="-funroll-loops"' \
 128         qmtest-g++
 129
 130    The compiler will then use the "-funroll-loops" switch when
 131    compiling.
 132
 133 4. If qmtest is not in your path, you can indicate the full path to
 134    QMTest by using the make variable "QMTEST_PATH", like this:
 135
 136       make QMTEST_PATH=/path/to/qmtest qmtest-g++
 137
 138 5. To start the QMTest GUI, use:
 139
 140       make qmtest-gui
 141
 142    (Note that this will run the program called "mozilla" in your path.
 143    If you want to use another browser, you must configure qmtest as
 144    described in its manual.)
 145
 146    Bear in mind that the QMTest GUI is insecure; malicious users with
 147    access to your machine may be able to run commands as if they were
 148    you.  The QMTest GUI only binds to the loopback IP addresss, which
 149    provides a measure of security, but not enough for use in untrusted
 150    environments.
 151
 152 6. If you have a multiprocessor, you can run the tests in parallel by
 153    passing the "-j" option to qmtest:
 154
 155       make QMTESTRUNFLAGS="-j 4" qmtest-g++
 156
 157    will run tests in four threads.  (It is also possible to run tests
 158    across multiple machines; for more information see the QMTest
 159    manual.)
 160
 161 7. If a test (say "g++.dg/abi/bitfield1.C") fails, and you want to get
 162    more detailed information, you can do:
 163
 164       cd qmtestsuite
 165       qmtest summarize g++.qmr g++.dg/abi/bitfield1.C
 166
 167    to get more information about the commands that were run and the
 168    output produced.