• A_guide_to_testcase_reduction

A Guide to Testcase Reduction

Our bug reporting instructions ask that a bug report include the preprocessed version of the file that triggers the bug. There are several methods to minimise a testcase. See the Gentoo guide on ICE reduction too.

This page ought to serve as an introduction to automatic testcase reduction using C-Vise tools or C-Reduce tools.

Simple ICE reduction

• We'll start with the most simple case, reducing the testcase for an internal compiler error (ICE). Start from the preprocessed input as got from using

  the -save-temps compiler option (the .i or .ii file), in the following we'll assume it is called testcase.i. As a first step remove empty lines and line/file markers inserted by the preprocessor

  > cat testcase.i | grep -v '^# .*$' | grep -v '^[[:space:]]*$' > testcase.x
  > mv testcase.x testcase.i

• The C-Vise (or C-Reduce) tool requires us to create a script that exits with status zero in case of the intermediate reduced testcase still is a testcase for what we got it in the first place (the same ICE is produced). A sample script may look like

  #!/bin/sh
  gcc -c -O -Wfatal-errors testcase.i 2>&1 | grep 'internal compiler error: in typeck.c:2534'

• Note the -Wfatal-errors option can greatly speed up reducing large testcases, because the compiler will not try error recovery and continue compiling invalid testcases. Of course if you start with an ICE-on-invalid testcase, this is not a good idea. You should be able to verify your script by invoking it with the unreduced testcase. Try if it has zero exit code. Now we can invoke C-Vise (or C-Reduce) to have it reduce the testcase using the script we just wrote

  > cvise check.sh testcase.i

• This will reduce the testcase until no single line can be removed from it without the check.sh script failing to identify it as a valid testcase.

The faster the script ('interestingness test'), the fastest the reduction will be:

• Avoid optimizing -O0 if it is a bug in the front-ends.

• If the bug is not triggered by a warning, disable all warnings -w.

• If the bug is an ICE that also happens with -fpermissive, then use it.

• If the bug is not in the preprocessor, use -fpreprocessed, otherwhise use -E.

• Use -S to avoid assembling.

• -o /dev/null if you do not need to analyze the output files.

• It may be slightly faster to invoke cc1 and cc1plus directly if the bug is not in the driver.

• For grepping, it is faster to not use regular expressions and invoke fgrep directly. It is also faster to use the option "--quiet".

Using C-Vise and C-Reduce

C-Vise and C-Reduce provide much faster reduction than the original delta tool. This reducer specifically targets C and C++ code and makes coordinated changes across the whole program: removing an array dimension, removing a function argument, reordering function calls, etc. C-Vise runs in parallel by default if multiple cores are available.

Reducing LTO bugs

• If you need to reduce testcases for LTO bugs that happen at link stage you have the problem that many object files (and thus source files) can be involved.
  • Now generate preprocessed source for the files required to build the remaining object files:
    • Go through the build log for each '.o' in the failing link command;
    • Find the command which created it, rerun that command with '-save-temps';
    • Replace the '.o' in the link command with the preprocessed source created (.i or .ii). Make sure to append any optimization options used in the command you found.
    • Repeat for all objects.
    You should be able to reproduce the ICE with

  gcc -r -nostdlib preprocessed-inputs rest-of-your-options

• (Note the use of "-r" to avoid link errors on undefined references.) And the preprocessed files can now be individually reduced with C-Vise. Be careful with creating invalid testcases here though. Safe reduction can be done on topflatform level 0 and level 0 with ignoring namespaces. This is to avoid differences in type layout for same types in different files.

  
  Alternatively C-Vise multi-file reduction may be used:

  cvise ./check.sh file1.ii file2.ii file3.ii ...

Optional: reduce the list of objects via cvise-delta

• If the bug is an internal compiler error you can bisect the files needed to trigger the bug using cvise-delta (provided by C-Vise package). From the lto1 invocation command extract the list-of-objects file (@/tmp/ccXXXXX) [use "-v -Wl,-debug" to find that line]; change spaces to newlines in there and prepend the current working directory name. Then use C-Vise (or C-Reduce) on that file with a check script similar to

/path/to/lto1 -o /dev/null @$1 rest-of-your-options 2>&1 | grep '...the ICE...'

• This will reduce the list of files (whatever argument you passed) needed as input, using response files (@x where x is a list of objects).

Legacy reduction tools

The Delta tool requires us to create a script that exits with status zero in case of the intermediate reduced testcase still is a testcase for what we got it in the first place (the same ICE is produced).

For Fortran, there is a patched version of Delta which takes subroutine/do/if boundaries into account.

A sample script may look like (the testcase filename is passed as argument to the script)

 #!/bin/sh
 gcc -c -O -Wfatal-errors $1 2>&1 | grep 'internal compiler error: in typeck.c:2534'

Note the -Wfatal-errors option can greatly speed up reducing large testcases, because the compiler will not try error recovery and continue compiling invalid testcases. Of course if you start with an ICE-on-invalid testcase, this is not a good idea.

You should be able to verify your script by invoking it with the unreduced testcase. Try if it has zero exit code.

Now we can invoke Delta to have it reduce the testcase using the script we just wrote

 > ~/bin/delta -test=check.sh -suffix=.i -cp_minimal=testcase-min.i testcase.i

This will reduce the testcase until no single line can be removed from it without the check.sh script failing to identify it as a valid testcase.

Using topformflat

The way delta reduces a testcase by removing complete lines often conflicts with the syntactic structure of a C/C++ testcase. To make testcase reduction faster and more accurate there exists the topformflat tool in the Delta distribution that puts syntactically related tokens on one line, thereby making it possible to, f.i. restrict reduction to whole-function removal in a first step. Basically you can control the nesting level up to which tokens are put to separate lines where a level of zero is all toplevel constructs onto a line on their own, level one would be each statement of a toplevel function on a separate line.

Reducing a big C++ testcase one usually starts with level zero, increasing it until Delta no longer can reduce the testcase further (due to the line-oriented reduction it may be worthwhile to start over with level zero again and iterate until there's no further reduction). An improved topformflat was posted at https://gcc.gnu.org/ml/gcc-patches/2005-08/msg01503.html where you can additionally specify if you want to ignore namespace and extern "C" as a nesting construct by specifying a second command line argument to topformflat.

 > ~/bin/topformflat 0 x < testcase.i > testcase.0x.i

Using multidelta

All the above can be simplified by using the multidelta tool that comes with the Delta distribution. The only differences is that the script should be able to be called without parameters. In the above example, the new script would be:

 #!/bin/bash
 TESTCASE=${1:-testcase.i}
 gcc -c -O -Wfatal-errors -w $TESTCASE 2>&1 | grep -q 'internal compiler error: in typeck.c:2534'

Further hints

• Sometimes it may be advisable to preserve parts of the testcase completely to make C-Vise (or C-Reduce) not reduce the testcase to nonsense still passing your script. This is also useful if you just want the smallest possible self-contained source for a function to ease analyzing whatever you are interested in. Split your testcase into two parts, one to be reduced and one invariant (called tail.i). Now we're going to reduce the first part under the presumption that the full testcase still compiles successfully.

 #!/bin/sh
 cat $1 ../../tail.i > x.i
 gcc -S x.i -Wfatal-errors

• You can even check for the same asm created in the above scheme, but you need to carefully create an ed script to cut out the relevant parts. An example could look like

 1,/^_Z8cpp_testRK5ArrayILi2Ed10BrickViewUES3_:/d
 /\.size/,$d
 ,w

• which deletes everything up to the interesting functions label and anything after the size marker after it. Create a sample asm file we can compare against with your initial testcase and the ed script (call it asm.s). Now the check script could look like:

 #!/bin/sh
 cat $1 ../../tail.i > x.i
 gcc -S x.i -Wfatal-errors
 if ! test "$?" = "0"; then
   exit 1
 fi
 ed x.s < ../../testcase.ed
 diff -u ../../asm.s x.s
 if ! test "$?" = "0"; then
   exit 1
 fi
 exit 0

Runtime bugs

Reducing "works with -O, doesn't work with -O2" type bugs

• First of all, you need to prepare the testcase so that you can easily tell, if it works or not works. The easiest way is to place some abort() in a conditional where you know it failed and ensure you return with zero exit code on success. For verifying if the reduced testcase is still correct, we need to use two compilations and runs, while watching out for possibly created infinite loops by the reduction process (adjust the timeout for your testcase).

  #!/bin/sh
  gcc -o works $1 -O -Wfatal-errors
  if ! test "$?" = "0"; then
    exit 1
  fi
  ( ulimit -t 10; ./works )
  if ! test "$?" = "0"; then
    exit 1
  fi
  gcc -o fails $1 -O2 -Wfatal-errors
  if ! test "$?" = "0"; then
    exit 1
  fi
  ( ulimit -t 10; ./fails )
  if test "$?" = "0"; then
    exit 1
  fi
  exit 0

• You can trivially make this script handle miscompilations that produce infinite loops or segmentation faults by adjusting the return value checks. Sometimes it may be impossible or hard to minimize a testcase when dealing with a miscompilation.

  See Analysing_Large_Testcases for a guide on reducing effort on analyzing miscompilations by using bisection scripts and GCC's debug counters.