Contents

Testing a different target from the current host
Testing with a simulator
Interpretation of testsuite results
1. Using validate_failures.py
Documentation on writing testcases
Running testsuite on Cygwin

GCC has its own testsuite which can be run after you compiled GCC see: http://gcc.gnu.org/install/test.html

Because the testsuite tends to be long-running, you may wish to increase the verbosity level so that you get feedback as to its progress. You can do this by adding "-v" to RUNTESTFLAGS. Each "-v" added increases the verbosity level by one, so specify it multiple times if you would like more output. Example:

make -k check-gcc RUNTESTFLAGS="-v -v"

This can be combined with the other RUNTESTFLAGS options mentioned in test.html above. Example:

make -k check-gcc RUNTESTFLAGS="compile.exp=2004* -v -v"

Testing a different target from the current host

In some circumstances it is possible to run tests locally for a different target than the current host (for example, darwin8 target tests on a darwin9 host system). In order to achieve this, you might also need to provide a 'sysroot' (to point at the libraries and headers for the target).

This can be accomplished by a command like this:

make -k check-gcc RUNTESTFLAGS="CFLAGS_FOR_TARGET=--sysroot=/path/to/target/root --target_board=unix/-other/-options"

You might wish to use:

CFLAGS_FOR_TARGET='$CFLAGS_FOR_TARGET --sysroot=/path/to/target/root'

if CFLAGS_FOR_TARGET is already set for your test case(s).

Testing with a simulator

For running the testsuite on a simulator (useful for cross targets) see: http://gcc.gnu.org/simtest-howto.html

and a similar page (with some useful links): Building_Cross_Toolchains_with_gcc

Interpretation of testsuite results

Normal testsuite results usually contain a few FAILs -- unexpected failures. Thus it might be hard to determine if the changes being tested actually broke something. There are several ways to deal with this situation:

The most common approach is to compare two testsuite runs, one with and one without the tested changes. Check with <code>diff</code> these two trees to make sure that you test only the differences you intend to test.
- There are several scripts in contrib/ that can be used to obtain condensed testresult differences:
- dg-cmp-results.sh
  - /src/gcc/contrib/dg-cmp-results.sh -v -v '*' ../gcc.orig/gcc/testsuite/gcc/gcc.sum gcc/testsuite/gcc/gcc.sum
    or something like
    for i in $(find gcc/ -name "*.sum");do ../../src/gcc/contrib/dg-cmp-results.sh -v -v '' ../gcc.orig/$i $i;done
Look in the http://gcc.gnu.org/ml/gcc-testresults/ for the mailed results for your platform that might serve as a baseline.
If you are developing a new port, aim for zero (or as few as possible and practical) FAILs.

Using validate_failures.py

The script <src>/contrib/testsuite-management/validate_failures.py can be used to maintain a list of known/expected failures outside of DejaGNU. This is useful when working in a branch with relatively stable failures, which you have determined to be "ignorable".

Since modifying dejagnu files to mark XFAILs is not always trivial, validate_failures.py offers a lightweight approach that can support lists for multiple targets.

The idea is to create a manifest file that contains the FAIL, XPASS, UNRESOLVED output from make check. You cut and paste that output into the manifest file and then use validate_failures.py to decide whether the failures are ignorable or not.

$ <src>/contrib/testsuite-management/validate_failures.py --help
Usage: This script provides a coarser XFAILing mechanism that requires no
detailed DejaGNU markings.  This is useful in a variety of scenarios:

- Development branches with many known failures waiting to be fixed.
- Release branches with known failures that are not considered
  important for the particular release criteria used in that branch.

The script must be executed from the toplevel build directory.  When
executed it will:

1- Determine the target built: TARGET
2- Determine the source directory: SRCDIR
3- Look for a failure manifest file in
   <SRCDIR>/contrib/testsuite-management/<TARGET>.xfail
4- Collect all the <tool>.sum files from the build tree.
5- Produce a report stating:
   a- Failures expected in the manifest but not present in the build.
   b- Failures in the build not expected in the manifest.
6- If all the build failures are expected in the manifest, it exits
   with exit code 0.  Otherwise, it exits with error code 1.


Options:
  -h, --help            show this help message and exit
  --build_dir=BUILD_DIR
                        Build directory to check (default = .)
  --manifest            Produce the manifest for the current build (default =
                        False)
  --force               When used with --manifest, it will overwrite an
                        existing manifest file (default = False)
  --verbosity=VERBOSITY
                        Verbosity level (default = 0)

Documentation on writing testcases

The testuite uses DejaGnu (GNU Testing Framework)

The DejaGnu manual can be found here: http://www.gnu.org/software/dejagnu/manual/

Giovanni Bajo posted a patch in gcc-patches in which he added the section How to prepare a testcase to contribute.html

Running testsuite on Cygwin

The testsuite runs on Cygwin, however slowly, and it is likely to hit a bug in Cygwin's process info management. To avoid that, anyone who wants to test on Cygwin is advised to build Cygwin DLL manually with this patch applied.