[PATCH 00/21] Add -fself-test framework for fast, early unit-testing (unittests v5)
David Malcolm
dmalcolm@redhat.com
Wed Jun 1 20:54:00 GMT 2016
This is effectively v5 of the unittests proposal; for the earlier
versions see:
* v1: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg00765.html
* v2: https://gcc.gnu.org/ml/gcc-patches/2015-06/msg01224.html
* v3: https://gcc.gnu.org/ml/gcc-patches/2015-10/msg02947.html
* v4: https://gcc.gnu.org/ml/gcc-patches/2015-11/msg02379.html
Bernd said (in https://gcc.gnu.org/ml/gcc-patches/2015-11/msg01981.html ):
> For some of the simpler infrastructure tests such as the ones in this
> patch kit (bitmap, vec or wide-int functionality testing and such),
> we had the idea of putting these into every ENABLE_CHECKING compiler,
> and run them after building stage1, controlled by a -fself-test flag.
> It's better to detect such basic failures early rather than complete
> a full bootstrap and test cycle. It also keeps the tests alongside the
> rest of the implementation, which I consider desirable for such
> relatively simple data structures."
So the main difference is this version of the patch kit is that the tests
are run much earlier: rather than have a DejaGnu test below gcc.dg that
compiles a dummy file with -fself-test, in this iteration, gcc/Makefile.in
is updated so that the selftests are run during the build.
The patch kit adds a "selftests" phony target to gcc/Makefile.in
ensuring that the selftests must be successfully run for the build to complete.
The time taken to run the testsuite is also printed.
A "s-selftests" file is created, so that the selftests need only be run
once each time "cc1" is rebuilt in a working copy.
They are run at each stage in a bootstrap. For example, looking at a
bootstrap log:
$ grep "fself-test:" test/experiment/x86_64-pc-linux-gnu/build/make.log
-fself-test: 522 pass(es); 0 failure(s) in 0.013000 seconds
-fself-test: 522 pass(es); 0 failure(s) in 0.006000 seconds
-fself-test: 522 pass(es); 0 failure(s) in 0.006000 seconds
shows that the selftests are indeed run at each stage of a bootstrap,
and that they're fast (well under a second).
There's also a debug target, so that we can type:
make selftests-gdb
to run the selftests under gdb.
I also patched things so that it aborts on the first failure, making
failures easy to track down in the debugger, though requiring us to
make the selftests robust.
Assuming that other people like these changes, it has a few
implications:
* selftests ought to be fast. We don't want to slow down the build.
* running -fself-test should be terse by default; we may want an option
for it to be more verbose
* poorly written selftests could break the build for other people
(e.g. accidentally making target-specific assumptions).
The patch kit adds some new tests:
* some self-tests for diagnostic-show-locus.c. This shows how to unit-test
an internal API.
* conversion of the Levenshtein selftest from a plugin to using
-fself-test (we could move various other tests from plugins to run
earlier).
There is a bug with GNU ld in which it doesn't run tests in source files
that purely contain tests (hence I was #include-ing these from toplev.c).
I tracked down the issue, which I've filed against GNU ld here:
https://sourceware.org/bugzilla/show_bug.cgi?id=20152
and a minimal reproducer can be seen here:
https://github.com/davidmalcolm/test-ctor
The issue is that these don't work with GNU ld if they're linked into
a .a file (libbackend.a in our case) and then linked from there into
cc1. They work if they're linked directly into cc1 without going through
an archive.
We could link these files directly into cc1, or we could move to
manual test registration instead.
Or we could not have registration, and simply hardcode what tests run,
and what order they run in.
Testing:
* successfully bootstrapped®ression tested on x86_64-pc-linux-gnu.
* successfully built with 193 configurations using
contrib/config-list.mk. The remaining configurations fail for
various reasons:
* 9 vxworks configurations fail to run the selftests with:
xgcc: fatal error: environment variable ‘WIND_BASE’ not defined
* 4 configurations fail due to pre-existing warnings: fails on 3
configurations (microblaze and rl78) due to warnings which I've
posted fixes for), and on avr-rtemsOPT-enable-obsolete due to a
different pair of warnings.
* 2 failures: rs6000-ibm-aix4.3 and rs6000-ibm-aix5.1.0 due to a
pre-existing bug on trunk (I've filed this as PR target/71375).
* I haven't yet tested this with build != host.
Patch 1 is the main part of the implementation; it's similar to v4.
Patch 2 adds the new Makefile.in integration.
Patch 3 reduces the verbosity of the tests by moving almost all logging
to a new optional logfile (without providing a way to set it at this
stage).
Patch 4 shows a way of filtering the tests using a command-line regex.
Patch 5 adds a new example test, showing how to unit-test an
implementation detail of diagnostic-show-locus.c
Patch 6 shows an example of porting one of our existing plugin-based
tests to be within -fself-test instead.
Patches 7-21 are the previously posted tests, albeit with some bug
fixes.
My recollection is that the major unresolved discussion areas in v4
of the patch kit were:
(a) do we want to stop running at the first failure, or to run the
testsuite to completion?
(b) should tests be automatically registered, or should this be manual?
(c) is this over-engineered; should we simply use gcc_assert etc?
For (a), this version of the patch kit switches to stopping at the
first failure.
For (b), this version of the patch kit stays with auto-registration.
For (c), this version stays with a "lite" version of GTest.
Thoughts? Does the "running tests early" approach suggest answers to
these?
David Malcolm (21):
Selftest framework (unittests v5)
Makefile.in integration
Various selftest::runner tweaks.
Add -fself-test-regex=
Add selftests for diagnostic-show-locus.c
Convert Levenshtein test from a plugin to a selftest
Add selftests to bitmap.c
Add selftests to tree-cfg.c
Add selftests to et-forest.c
Add selftests to fold-const.c
Add function-tests.c
Fix warning in function-tests.c
Fixup to function-tests.c
Remove x86_64-isms in function-tests.c
Add selftests to gimple.c
Add hash-map-tests.c
Add hash-set-tests.c
Add selftests to input.c
Add rtl-tests.c
Add selftests to tree.c
Add selftests to vec.c
gcc/Makefile.in | 26 +-
gcc/bitmap.c | 92 ++++
gcc/common.opt | 8 +
gcc/diagnostic-show-locus.c | 113 ++++
gcc/et-forest.c | 99 ++++
gcc/fold-const.c | 66 +++
gcc/function-tests.c | 629 +++++++++++++++++++++++
gcc/gimple.c | 103 ++++
gcc/hash-map-tests.c | 81 +++
gcc/hash-set-tests.c | 57 ++
gcc/input.c | 110 ++++
gcc/rtl-tests.c | 98 ++++
gcc/selftest.c | 210 ++++++++
gcc/selftest.h | 267 ++++++++++
gcc/spellcheck.c | 48 ++
gcc/testsuite/gcc.dg/plugin/levenshtein-test-1.c | 9 -
gcc/testsuite/gcc.dg/plugin/levenshtein_plugin.c | 64 ---
gcc/testsuite/gcc.dg/plugin/plugin.exp | 1 -
gcc/toplev.c | 55 ++
gcc/toplev.h | 2 +
gcc/tree-cfg.c | 264 ++++++++++
gcc/tree.c | 47 ++
gcc/vec.c | 142 +++++
23 files changed, 2512 insertions(+), 79 deletions(-)
create mode 100644 gcc/function-tests.c
create mode 100644 gcc/hash-map-tests.c
create mode 100644 gcc/hash-set-tests.c
create mode 100644 gcc/rtl-tests.c
create mode 100644 gcc/selftest.c
create mode 100644 gcc/selftest.h
delete mode 100644 gcc/testsuite/gcc.dg/plugin/levenshtein-test-1.c
delete mode 100644 gcc/testsuite/gcc.dg/plugin/levenshtein_plugin.c
--
1.8.5.3
More information about the Gcc-patches
mailing list