Preparing Testcases

See https://gcc.gnu.org/onlinedocs/gccint/Test-Directives.html#Test-Directives for official documentation. See also TestCaseWriting for another test-case howto.

Every patch submitted for approval should always come with a testcase to be added to the testsuite (there are a few exceptions that are listed below). The testcase should be able to test the functionality of the new features you add or to reproduce the bug on all affected platforms, and it is used to make sure that the bug stays fixed.

There are some basic guidelines you should follow while preparing a testcase: 

#include <stdlib.h>
void foo(void)
{
  abort();
}

it is better to write: 

extern void abort(void);
void foo(void)
{
  abort();
}

C++ testcases can prepend extern "C" while declaring such functions.

Exceptions are the C headers that GCC includes, such as <limits.h> and <stddef.h>. For feature testcases these are fine to include. Testcases including <limits.h> and then using macros such as INT_MAX are arguably more readable than ones using the predefined _INT_MAX_ directly.

There are a few excuses for not preparing a testcase.

Choose carefully which directory to place the test in.

Each language has its own directory, and often two directories: one for torture tests, and the other for normal tests. A torture test is a testcase which is compiled and optionally executed with many different optimization levels, and it is meant to stress GCC optimizers. Normal tests instead are meant to test for frontend features (which are not affected by optimization levels), or tests which need specific command line options and thus are not suited for the torture testsuite.

NOTES (these apply only to C testcases):

To add and test your testcase, it is sufficient to drop it into one of the above directories, and then run one of the below commands from the build directory 

# Generic version.
make check RUNTESTFLAGS=testsuitename.exp=testcasename.ext

# C++ only
make -C gcc check-g++ RUNTESTFLAGS=testsuitename.exp=testcasename.ext

testsuitename.exp is the name of the testsuite module (look for a file with extension .exp which is the closest, directory-wise, to the testcase you added). If the testcase does not work as expected, you can inspect the testsuite log, which is generated somewhere deep in the build directory (the logs for the GCC core can be found in <build-dir>/gcc/testsuite/*.log, while the logs for multilib libraries are at <build-dir>/<target>/<library>/testsuite/*.log, for instance build/i686-pc-linux-gnu/libstdc++-v3/testsuite/libstdc++.log).

The testsuite uses DejaGnu as its framework. The framework parses the testcases looking for special commands, written in comments, which describe the tests that must be performed. Each testcase might have several different tests performed on it, each one described by a command.

Each command issues a specific test to be done. The possible outcomes are the following

The testsuite summary lists only the tests that exited with FAIL or XPASS status, since the others just behaved as expected. In the former case, it means that the test failed, so your patch probably broke it. In the latter case, it means that your patch just fixed a bug which was already present in the testcase but marked as an expected failure. This means that a new testcase is probably not necessary, and your patch can be updated so that it also removes the xfail modifier from the existing testcase.

Test commands

The main commands that can be used are the following:

{ dg-do <action> }

{ dg-options <opts> }

{ dg-error "<match-regexp>" "<comment>" <{ target *-*-* }> <line-spec> }

{ dg-warning "<match-regexp>" "<comment>" <{ target *-*-* }> <line-spec> }

{ dg-excess-errors "<match-regexp>" }

{ dg-bogus "<match-regexp>" "<comment>" <{ target *-*-* }> <line-spec> }

Behaves like dg-error but with inverted logic: it tests for an error matching a certain regular expression not to be emitted on the line matching the <line-spec>. This is useful for situations where we want to make sure that a certain error message or warning is not emitted any more.

One could ask why to bother at all with dg-bogus . In fact, if we want an error not to be emitted while compiling the testcase, it is sufficient to do nothing at all, were the error emitted, the testcase compilation (as specified by dg-do ) would fail for excess errors. 'dg-bogus is mostly useful in combination with the xfail modifier (see below for an example).

{ dg-final { <final-action> } }

{ dg-final { scan-assembler "<match-regexp>" } }

{ dg-final { scan-assembler-not "<match-regexp>" } }

{ dg-final { scan-file "<filename>" "<match-regexp>" } }

{ dg-final { scan-file-not "<filename>" "<match-regexp>" } }

{ dg-final { scan-tree-dump-times "<match-regexp>" <num-expected-matches> "<dump-name>" } }

{ dg-final { scan-tree-dump "<match-regexp>" "<dump-name>" } }

{ dg-final { scan-tree-dump-not "<match-regexp>" "<dump-name>" } }

{ dg-output "<match-regexp>" }

{ <modifier> }

Common modifiers used in the testsuite

{ target "<triplets>" }

{ xfail "<triplets>" }

PS:

This text was taken from a patch from GiovanniBajo and a README, the comments from Joseph S. Myers and formatted for this Wiki by MichaelCieslinski.

TODO

The testsuite cannot detect duplicate error/warnings PR 30612, you need to use the syntax: 

/* { dg-bogus "message.*message" } */
/* { dg-warning "message" "" { target *-*-* } 1 } */

None: HowToPrepareATestcase (last edited 2016-09-28 19:33:35 by MartinSebor)