QMTest and the G++ testsuite

[Mark Mitchell <mark at codesourcery dot com>]

I am about to check in a patch on the mainline to allow the use of
QMTest -- a free testing tool developed by CodeSourcery -- instead of
DejaGNU, to run the G++ regression test suite.  (For the patch, see
gcc-patches.)

My objectives are:

1. To convert the entire GCC testsuite to QMTest.

2. To obtain consensus in the community that we should use QMTest as
   our testing tool, rather than DejaGNU.

We believe there are several advantages to QMTest, vis a vis DejaGNU,
including:

1. Support for parallel testing; multiple test cases can be run at
   once, thereby speeding up testing on multiprocessor machines (and
   even uniprocessor machines, since our testing is I/O bound).

2. Support for direct comparisons between test results.  QMTest can
   easily answer the question "Did I break anything relative to the
   results I have from yesterday?"

I realize that realizing the objectives above will take a fair amount
of work, and that it may not succeed at all.  If QMTest isn't able to
do a better job than DejaGNU, it certainly will not.

QMTest is a young tool, and there are doubtless going to be issues:
QMTest may not work on some platforms, or there may be bugs that
affect usability, or ...

Let's try to keep the discussion focused on how to get results with
QMTest that are at least as good as we get with DejaGNU.  If we use
this as a forum for talking about everyone's personal viewpoint on
testing on general and "I've always really wanted to see X ability",
we're going to get too distracted.

I will be willing to do the work required to get something that people
are happy with -- both in QMTest and in the extension classes that
handle the GCC testsuite -- within reason.  I would appreciate help in
trying out QMTest and delineating what, if any, improvements are
needed before we can move to QMTest instead of DejaGNU.  I would also
appreciate people making as few changes as possible to the DejaGNU
harness since I'll be having to keep up on the QMTest side.  (I'll
probably make some changes to the testsuite as time goes by to try to
make things more uniform; that will be a win no matter what tool we
use.  There's really no reason for us to have so many different ways
of doing things in the testsuite.)

I think the first step is to try get a consensus that it is OK to do
your G++ regression-testing with QMTest, instead of DejaGNU, before
check-in.  I've verified that I get exactly the same results with
DejaGNU and with QMTest.  If we can agree that it's OK to use either
tool, that will give us a transition period where we can iron out the
issues that remain.

I've attached the file README.QMTEST, now present in the "testsuite"
directory, that explains how to try out "make qmtest-g++" instead of
"make check-g++".

--
Mark Mitchell                   mark@codesourcery.com
CodeSourcery, LLC               http://www.codesourcery.com

Testing with QMTest
===================

You can use QMTest to test G++.  (In the future, it may be possible to
test other parts of GCC with QMTest as well, but it is not possible
yet.)

The use of QMTest to run the G++ tests has not been approved as an
officially supported testing procedure.  Therefore, you must run the
tests using DejaGNU (with "make check-g++") before committing changes
that affect G++.

Differences from DejaGNU
========================

Although QMTest can be used to run the G++ test suite, it works
somewhat differently from DejaGNU.  In particular:

- In DejaGNU, a single source file contains many tests.  Each
  line where a diagnostic is expected is considered a separate
  test.  Testing for successful compilation and testing for
  successful execution of the generated program are considered
  separate tests.  Thus, a test "test.C" could contain, say,
  seven tests; some of which might pass and some of which might
  fail.

  With QMTest, each source file is considered a single test.  If any
  of the seven sub-tests fail, the entire test is considered to fail.
  However, QMTest does present information about *why* the test
  failed, so the same information is effectively available.

  It is true that, therefore, causing an already failing test to "fail
  more" is not immediately detectable through an additional unexpected
  failure messages when using QMTest.  On the other hand, most people
  seem to think of each source file as "a test", not "twelve tests",
  so the model QMTest uses may be more natural.

- In DejaGNU, tests themselves keep track of expected and unexpected
  failures.  The QMTest philosophy is that expected failures should be
  stored separately from the tests themselves; in particular, that
  tonights results can be tomorrow's expectations.  In order to
  preserve compatibility with DejaGNU, the first time you use QMTest
  to test G++, QMTest computes the set of expected failures indicated
  by the tests, and then compares the actual results with these
  results.  Therefore, if you change the expected failure notations in
  the DejaGNU tests, you must rebuild the set of expected failures.

  To do this, remove the file "qmtestsuite/gpp-expected.qmr".  Then,
  when you rerun the tests, the expected failures will be
  automatically recalculated.

Setting Up
==========

You must download and install the following software:

- Python 2.1 (or greater)

  See http://www.python.org.

  You may already have Python on your system; in particular, many
  GNU/Linux systems ship with Python installed.

  Installation instructions are available on the web-site.

- QMTest 1.1.4 (or greater)

  See http://www.qmtest.com.

  QMTest is available at:

    http://www.codesourcery.com/qm/qmtest_download

  or:

    ftp://ftp.codesourcery.com/pub/qmtest

  Installation instructions are available on the web-site.

- QMTC 1.1

  This package is available from:

  ftp://ftp.codesourcery.com/pub/qmtest/qmtest/qmtc/qmtc-<version>.tar.gz

  See the file called INSTALL in the distribution.

Running the Tests
=================

To run the tests, run "make qmtest-g++" in the gcc directory of your
build tree.  The first time that you do this, QMTest will calculate
the set of tests that are expected to fail on your platform, so it
will take several minutes before you see any test results.  After the
first time, QMTest will start running the tests much more quickly.

If the test summary printed at the test run indicates no unexpected
failures, then G++ is behaving as expected on your target.  (Some
unexpected passes are normal.)

You can obtain detailed information about why tests failed in one
of two ways:

1. By invoking QMTest with the "-f full" option.  For example:

     make QMTESTRUNFLAGS="-f full" qmtest-g++

2. Examining the log file qmtestsuite/gpp.qmr after the tests have
   run.

Here are some more advanced usage instructions:

1. To run a particular set of tests (rather than all of the tests),
   use the make variable "QMTEST_GPP_TESTS".  For example,

     make QMTEST_GPP_TESTS="gpp.dg" qmtest-g++

   will run only the tests in the g++.dg subdirectory, and:

     make QMTEST_GPP_TESTS="gpp.dg.special.conpr1 \
                            gpp.old-deja.other.access2" \
          qmtest-g++

   will run only the two tests indicated.

2. To run qmtest with particular flags, use the make variables
   "QMTESTFLAGS" and "QMTESTRUNFLAGS".  For example:

      make QMTESTFLAGS="-v" QMTESTRUNFLAGS="-f full" qmtest-g++

   will run qmtest like this:

      qmtest -v run -f full ...

3. To run the compiler with particular flags, use QMTESTRUNFLAGS to
   set the QMTest context variable "GPPTest.flags", like this:

      make QMTESTRUNFLAGS='-c GPPTest.flags="-funroll-loops"' qmtest-g++

   The compiler will then use the "-funroll-loops" switch when
   compiling.

4. If qmtest is not in your path, you can indicate the full path to
   QMTest by using the make variable "QMTEST_PATH", like this:

      make QMTEST_PATH=/path/to/qmtest qmtest-g++

5. To start the QMTest GUI, use:

      make qmtest-gui

   (Note that this will run the program called "netscape" in your path.
   If you want to use another browser, you must configure qmtest as
   described in its manual.)

   Bear in mind that the QMTest GUI is insecure; malicious users with
   access to your machine may be able to run commands as if they were
   you.  The QMTest GUI only binds to the loopback IP addresss, which
   provides a measure of security, but not enough for use in untrusted
   environments.

6. If you have a multiprocessor, you can run the tests in parallel by
   passing the "-j" option to qmtest:

      make QMTESTRUNFLAGS="-j 4" qmtest-g++

   will run tests in four threads.  (It is also possible to run tests
   across multiple machines; for more information see the QMTest
   manual.)

7. If you are testing a cross compiler, you must specify an interpreter
   that is capable of running the generated program.  It must be a
   program "p" such that:

      p program arg1 arg2 arg3 ...

   behaves exactly like running:

      program arg1 arg2 arg3 ...

   would on the target machine.  You specify this program via the
   "CompilerTest.interpreter" context variable:

      make QMTESTRUNFLAGS='-c CompilerTest.interpreter=/path/to/interpreter'
           qmtest-g++