A Guide to Testcase Reduction

Our bug reporting instructions ask that a bug report include the preprocessed version of the file that triggers the bug. There are several methods to minimise a testcase.

This page ought to serve as an introduction to automatic testcase reduction using C-Vise tools or C-Reduce tools.

Simple ICE reduction

  > cat testcase.i | grep -v '^# .*$' | grep -v '^[[:space:]]*$' > testcase.x
  > mv testcase.x testcase.i

  #!/bin/sh
  gcc -c -O -Wfatal-errors testcase.i 2>&1 | grep 'internal compiler error: in typeck.c:2534'

  > cvise check.sh testcase.i

The fastest is this script, the fastest the reduction will be:

Using C-Vise and C-Reduce

C-Vise and C-Reduce provide much faster reduction than the original delta tool. This reducer specifically targets C and C++ code and makes coordinated changes across the whole program: removing an array dimension, removing a function argument, reordering function calls, etc. C-Vise runs in parallel by default if multiple cores are available.

Reducing "works with -O, doesn't work with -O2" type bugs

  #!/bin/sh
  gcc -o works $1 -O -Wfatal-errors
  if ! test "$?" = "0"; then
    exit 1
  fi
  ( ulimit -t 10; ./works )
  if ! test "$?" = "0"; then
    exit 1
  fi
  gcc -o fails $1 -O2 -Wfatal-errors
  if ! test "$?" = "0"; then
    exit 1
  fi
  ( ulimit -t 10; ./fails )
  if test "$?" = "0"; then
    exit 1
  fi
  exit 0

Reducing LTO bugs

  #!/bin/sh
  /path/to/lto1 -o /dev/null @$1 rest-of-your-options 2>&1 | grep '...the ICE...'

  gcc -r -nostdlib preprocessed-inputs rest-of-your-options

  cvise ./check.sh file1.ii file2.ii file3.ii ...

Legacy reduction tools

The Delta tool requires us to create a script that exits with status zero in case of the intermediate reduced testcase still is a testcase for what we got it in the first place (the same ICE is produced).

For Fortran there is a patched version of Delta which takes subroutine/do/if boundaries into account.

A sample script may look like (the testcase filename is passed as argument to the script)

 #!/bin/sh
 gcc -c -O -Wfatal-errors $1 2>&1 | grep 'internal compiler error: in typeck.c:2534'

Note the -Wfatal-errors option can greatly speed up reducing large testcases, because the compiler will not try error recovery and continue compiling invalid testcases. Of course if you start with an ICE-on-invalid testcase, this is not a good idea.

You should be able to verify your script by invoking it with the unreduced testcase. Try if it has zero exit code.

Now we can invoke Delta to have it reduce the testcase using the script we just wrote

 > ~/bin/delta -test=check.sh -suffix=.i -cp_minimal=testcase-min.i testcase.i

This will reduce the testcase until no single line can be removed from it without the check.sh script failing to identify it as a valid testcase.

Using topformflat

The way delta reduces a testcase by removing complete lines often conflicts with the syntactic structure of a C/C++ testcase. To make testcase reduction faster and more accurate there exists the topformflat tool in the Delta distribution that puts syntactically related tokens on one line, thereby making it possible to, f.i. restrict reduction to whole-function removal in a first step. Basically you can control the nesting level up to which tokens are put to separate lines where a level of zero is all toplevel constructs onto a line on their own, level one would be each statement of a toplevel function on a separate line.

Reducing a big C++ testcase one usually starts with level zero, increasing it until Delta no longer can reduce the testcase further (due to the line-oriented reduction it may be worthwhile to start over with level zero again and iterate until there's no further reduction). An improved topformflat was posted at https://gcc.gnu.org/ml/gcc-patches/2005-08/msg01503.html where you can additionally specify if you want to ignore namespace and extern "C" as a nesting construct by specifying a second command line argument to topformflat.

 > ~/bin/topformflat 0 x < testcase.i > testcase.0x.i

Using multidelta

All the above can be simplified by using the multidelta tool that comes with the Delta distribution. The only differences is that the script should be able to be called without parameters. In the above example, the new script would be:

 #!/bin/bash
 TESTCASE=${1:-testcase.i}
 gcc -c -O -Wfatal-errors -w $TESTCASE 2>&1 | grep -q 'internal compiler error: in typeck.c:2534'

Further hints

 #!/bin/sh
 cat $1 ../../tail.i > x.i
 gcc -S x.i -Wfatal-errors

 1,/^_Z8cpp_testRK5ArrayILi2Ed10BrickViewUES3_:/d
 /\.size/,$d
 ,w

 #!/bin/sh
 cat $1 ../../tail.i > x.i
 gcc -S x.i -Wfatal-errors
 if ! test "$?" = "0"; then
   exit 1
 fi
 ed x.s < ../../testcase.ed
 diff -u ../../asm.s x.s
 if ! test "$?" = "0"; then
   exit 1
 fi
 exit 0

None: A_guide_to_testcase_reduction (last edited 2020-08-18 13:05:28 by TobiasBurnus)