Our current test suite is rather coarse-grained, acting at the level
of running gcc subprocesses, and verifying externally-visible
properties:
- Did the compile succeed?
- Was a particular dumpfile emitted, containing something matching
some regex?
- Does the generated code run and exit gracefully?
etc
The strength of this approach is that we have good "integration"
coverage: we know that the compiler runs and generates good code.
However it's slow, and doesn't allow very precise control over what
properties we can assert.
The following patch kit adds a suite of unit tests to gcc, aimed at
giving test coverage for properties that are difficult to test for
in the current system. For example, there are tests of ggc to
verify that gengtype is doing sane things, of various container
classes (vec, hash_map, hash_set) and of wide-int. Some of the
tests are rather "placeholdery" e.g. the tests of folding trees,
where there's plenty of room for adding new testcases.
I've split them up into multiple patches for ease of review, but
they all stand together.
I picked the Google Test framework:
http://code.google.com/p/googletest/
I didn't do a very thorough survey of C++ test frameworks; I picked
this one as it's used by several very large projects (including
LLVM [1]), and is actively maintained. Working with it has largely been
a pleasant experience: it generates good error messages when tests
fail (e.g. with enough information so that I can click on failures
in Emacs and have it go to the failing test), and is sane to work
with in gdb. The log is easy to read; I've added an example to
the end of this mail. It supports parameterizing testcases across
multiple types (I use this for testing wide-int).
The only other "framework" I've used has been the DejaGnu unittest
header, which I use for the jit testsuite; I can't recommend it (it's
a pain to use, and I've had to extend it repeatedly to get basic
functionality like string equality assertions).
The testsuite is intended to be very fast, all in one process, and it
takes less than a second to run; it fact, the time is dominated by a
single very slow test, which takes 300-400ms to run, but which could
be sped up [2]; other than that, it takes about 10ms to run.
Structurally, the patches add a "unittests" frontend: this is rather
analogous to libgccjit.so: it's a dummy frontend. The
unittests/Make-lang.in builds a libgccunittests.so, which does
nothing, stubbing out the frontend hooks, but provides a DSO holding
the code to be tested.
libgccunittests.so is then linked into a "unittests.exe" binary, which
holds the actual testcases. An advantage of this separation is that
although linking libgccunittests.so is rather slow (like linking "cc1"
etc), unittests.exe is very fast to link, so that it's very fast to
hack on individual tests without needing to relink "everything" for
each edit.
Of course, this means that we can't unittest the real frontends this
way (but we couldn't before, and maybe we can find a route to doing
this).
I have the Make-lang.in implementing the "unittests" target so that
it builds and *runs* the unit tests i.e. if you do a "make", the
unittests are run (rather than just on a "make check"). Given how
fast they are (especially relative to "make check", the only issue
I can see with this is the output log spew). One nice thing about
doing it there is that it can be run at each stage of a bootstrap,
so hopefully we fail earlier when we're going to fail.
I marked it in config-lang.in as
build_by_default="no"
so you have to opt-in to building it by adding "unittests" to the
--enable-languages=
configure options.
The split of the bulk of the link into a libgccjitunittests.so
means that it also requires the
--enable-host-shared
configure-time option.
It doesn't yet bootstrap; the link fails with:
test-folding.o: In function `testing::AssertionResult testing::internal::CmpHelperNE<tree_node*, tree_node*>(char const*, char const*, tree_node* const&, tree_node* const&)':
test-folding.c:(.text._ZN7testing8internal11CmpHelperNEIP9tree_nodeS3_EENS_15AssertionResultEPKcS6_RKT_RKT0_[_ZN7testing8internal11CmpHelperNEIP9tree_nodeS3_EENS_15AssertionResultEPKcS6_RKT_RKT0_]+0x26e): undefined reference to `testing::internal::StringStreamToString(std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*)'
though this was using a system copy of gtest
(is this due to the C++ stdlib ABI change? it seems to only affect
tests using EXPECT_STREQ, and if I hack them out, it seems to work).
(perhaps we'd need to bundle our own copy of gtest?)
Here's a sample log (and it's sanely color-coded if you run it at
a tty):
$ make unittests
[==========] Running 56 tests from 16 test cases.
[----------] Global test environment set-up.
[----------] 6 tests from ggc_test
[ RUN ] ggc_test.tree_marking
[ OK ] ggc_test.tree_marking (0 ms)
[ RUN ] ggc_test.custom_struct
[ OK ] ggc_test.custom_struct (0 ms)
[ RUN ] ggc_test.finalization
[ OK ] ggc_test.finalization (0 ms)
[ RUN ] ggc_test.deletable_global
[ OK ] ggc_test.deletable_global (1 ms)
[ RUN ] ggc_test.inheritance
[ OK ] ggc_test.inheritance (0 ms)
[ RUN ] ggc_test.chain_next
[ OK ] ggc_test.chain_next (326 ms)
[----------] 6 tests from ggc_test (327 ms total)
[----------] 5 tests from bitmap_test
[ RUN ] bitmap_test.gc_alloc
[ OK ] bitmap_test.gc_alloc (0 ms)
[ RUN ] bitmap_test.set_range
[ OK ] bitmap_test.set_range (0 ms)
[ RUN ] bitmap_test.clear_bit_in_middle
[ OK ] bitmap_test.clear_bit_in_middle (0 ms)
[ RUN ] bitmap_test.copying
[ OK ] bitmap_test.copying (0 ms)
[ RUN ] bitmap_test.bitmap_single_bit_set_p
[ OK ] bitmap_test.bitmap_single_bit_set_p (0 ms)
[----------] 5 tests from bitmap_test (0 ms total)
[----------] 3 tests from cfg_test
[ RUN ] cfg_test.linear_chain
[ OK ] cfg_test.linear_chain (1 ms)
[ RUN ] cfg_test.diamond
[ OK ] cfg_test.diamond (0 ms)
[ RUN ] cfg_test.fully_connected
[ OK ] cfg_test.fully_connected (0 ms)
[----------] 3 tests from cfg_test (1 ms total)
[----------] 1 test from tree_folding_test
[ RUN ] tree_folding_test.arithmetic_folding
[ OK ] tree_folding_test.arithmetic_folding (0 ms)
[----------] 1 test from tree_folding_test (0 ms total)
[----------] 2 tests from function_test
[ RUN ] function_test.fndecl_int_void
[ OK ] function_test.fndecl_int_void (0 ms)
[ RUN ] function_test.fndecl_float_intchar
[ OK ] function_test.fndecl_float_intchar (0 ms)
[----------] 2 tests from function_test (0 ms total)
[----------] 4 tests from representation_test
[ RUN ] representation_test.gimplification
[ OK ] representation_test.gimplification (0 ms)
[ RUN ] representation_test.building_cfg
[ OK ] representation_test.building_cfg (0 ms)
[ RUN ] representation_test.conversion_to_ssa
[ OK ] representation_test.conversion_to_ssa (0 ms)
[ RUN ] representation_test.expansion_to_rtl
[ OK ] representation_test.expansion_to_rtl (6 ms)
[----------] 4 tests from representation_test (6 ms total)
[----------] 5 tests from gimple_test
[ RUN ] gimple_test.assign_single
[ OK ] gimple_test.assign_single (0 ms)
[ RUN ] gimple_test.assign_binop
[ OK ] gimple_test.assign_binop (0 ms)
[ RUN ] gimple_test.nop_stmt
[ OK ] gimple_test.nop_stmt (0 ms)
[ RUN ] gimple_test.return_stmt
[ OK ] gimple_test.return_stmt (0 ms)
[ RUN ] gimple_test.return_without_value
[ OK ] gimple_test.return_without_value (0 ms)
[----------] 5 tests from gimple_test (0 ms total)
[----------] 1 test from hash_map_test
[ RUN ] hash_map_test.map_of_strings_to_int
[ OK ] hash_map_test.map_of_strings_to_int (0 ms)
[----------] 1 test from hash_map_test (0 ms total)
[----------] 1 test from hash_set_test
[ RUN ] hash_set_test.set_of_strings
[ OK ] hash_set_test.set_of_strings (0 ms)
[----------] 1 test from hash_set_test (0 ms total)
[----------] 4 tests from location_test
[ RUN ] location_test.accessing_ordinary_linemaps
[ OK ] location_test.accessing_ordinary_linemaps (0 ms)
[ RUN ] location_test.unknown_location
[ OK ] location_test.unknown_location (0 ms)
[ RUN ] location_test.builtins
[ OK ] location_test.builtins (0 ms)
[ RUN ] location_test.reading_source_line
[ OK ] location_test.reading_source_line (0 ms)
[----------] 4 tests from location_test (0 ms total)
[----------] 2 tests from rtl_test
[ RUN ] rtl_test.test_single_set
[ OK ] rtl_test.test_single_set (0 ms)
[ RUN ] rtl_test.uncond_jump
[ OK ] rtl_test.uncond_jump (0 ms)
[----------] 2 tests from rtl_test (0 ms total)
[----------] 3 tests from tree_test
[ RUN ] tree_test.integer_constants
[ OK ] tree_test.integer_constants (0 ms)
[ RUN ] tree_test.identifiers
[ OK ] tree_test.identifiers (0 ms)
[ RUN ] tree_test.labels
[ OK ] tree_test.labels (0 ms)
[----------] 3 tests from tree_test (0 ms total)
[----------] 10 tests from vec_test
[ RUN ] vec_test.quick_push
[ OK ] vec_test.quick_push (0 ms)
[ RUN ] vec_test.safe_push
[ OK ] vec_test.safe_push (0 ms)
[ RUN ] vec_test.truncate
[ OK ] vec_test.truncate (0 ms)
[ RUN ] vec_test.safe_grow_cleared
[ OK ] vec_test.safe_grow_cleared (0 ms)
[ RUN ] vec_test.pop
[ OK ] vec_test.pop (0 ms)
[ RUN ] vec_test.safe_insert
[ OK ] vec_test.safe_insert (0 ms)
[ RUN ] vec_test.ordered_remove
[ OK ] vec_test.ordered_remove (0 ms)
[ RUN ] vec_test.unordered_remove
[ OK ] vec_test.unordered_remove (0 ms)
[ RUN ] vec_test.block_remove
[ OK ] vec_test.block_remove (0 ms)
[ RUN ] vec_test.qsort
[ OK ] vec_test.qsort (0 ms)
[----------] 10 tests from vec_test (1 ms total)
[----------] 3 tests from wide_int_test/0, where TypeParam = <type>
[ RUN ] wide_int_test/0.test_printing
[ OK ] wide_int_test/0.test_printing (0 ms)
[ RUN ] wide_int_test/0.test_ops
[ OK ] wide_int_test/0.test_ops (0 ms)
[ RUN ] wide_int_test/0.test_comparisons
[ OK ] wide_int_test/0.test_comparisons (0 ms)
[----------] 3 tests from wide_int_test/0 (0 ms total)
[----------] 3 tests from wide_int_test/1, where TypeParam = <type>
[ RUN ] wide_int_test/1.test_printing
[ OK ] wide_int_test/1.test_printing (0 ms)
[ RUN ] wide_int_test/1.test_ops
[ OK ] wide_int_test/1.test_ops (0 ms)
[ RUN ] wide_int_test/1.test_comparisons
[ OK ] wide_int_test/1.test_comparisons (0 ms)
[----------] 3 tests from wide_int_test/1 (0 ms total)
[----------] 3 tests from wide_int_test/2, where TypeParam = <type>
[ RUN ] wide_int_test/2.test_printing
[ OK ] wide_int_test/2.test_printing (0 ms)
[ RUN ] wide_int_test/2.test_ops
[ OK ] wide_int_test/2.test_ops (0 ms)
[ RUN ] wide_int_test/2.test_comparisons
[ OK ] wide_int_test/2.test_comparisons (0 ms)
[----------] 3 tests from wide_int_test/2 (0 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 16 test cases ran. (338 ms total)
[ PASSED ] 56 tests.
Thoughts?
Dave
[1] though it's not clear to me if LLVM is actually still using it;
see e.g. http://blog.llvm.org/2009/12/lit-it.html
[2] the test of chain_next/chain_prev in test-ggc.c, which needed
a very large linked list in order to reliably overflow the stack on
my box; perhaps this could be eliminated by adding something to
libiberty to shrink the stack size?
David Malcolm (17):
Add Make-lang.in and config-lang.in to gcc/unittests
Add test-bitmap.c to gcc/unittests
Add test-cfg.c to gcc/unittests
Add test-folding.c to gcc/unittests
Add test-functions.c to gcc/unittests
Add test-ggc.c to gcc/unittests
Add test-gimple.c to gcc/unittests
Add test-hash-map.c to gcc/unittests
Add test-hash-set.c to gcc/unittests
Add test-locations.c to gcc/unittests
Add test-rtl.c to gcc/unittests
Add test-tree.c to gcc/unittests
Add test-vec.c to gcc/unittests
Add test-wide-int.c to gcc/unittests
Add unittests-frontend.c to gcc/unittests
Add unittests-main.c to gcc/unittests
toplev.c: move location_adhoc_data_fini call
gcc/toplev.c | 3 +-
gcc/unittests/Make-lang.in | 200 ++++++++++++
gcc/unittests/config-lang.in | 34 ++
gcc/unittests/test-bitmap.c | 117 +++++++
gcc/unittests/test-cfg.c | 319 +++++++++++++++++++
gcc/unittests/test-folding.c | 121 +++++++
gcc/unittests/test-functions.c | 634 +++++++++++++++++++++++++++++++++++++
gcc/unittests/test-ggc.c | 292 +++++++++++++++++
gcc/unittests/test-gimple.c | 179 +++++++++++
gcc/unittests/test-hash-map.c | 78 +++++
gcc/unittests/test-hash-set.c | 54 ++++
gcc/unittests/test-locations.c | 148 +++++++++
gcc/unittests/test-rtl.c | 94 ++++++
gcc/unittests/test-tree.c | 102 ++++++
gcc/unittests/test-vec.c | 162 ++++++++++
gcc/unittests/test-wide-int.c | 186 +++++++++++
gcc/unittests/unittests-frontend.c | 250 +++++++++++++++
gcc/unittests/unittests-main.c | 108 +++++++
18 files changed, 3080 insertions(+), 1 deletion(-)
create mode 100644 gcc/unittests/Make-lang.in
create mode 100644 gcc/unittests/config-lang.in
create mode 100644 gcc/unittests/test-bitmap.c
create mode 100644 gcc/unittests/test-cfg.c
create mode 100644 gcc/unittests/test-folding.c
create mode 100644 gcc/unittests/test-functions.c
create mode 100644 gcc/unittests/test-ggc.c
create mode 100644 gcc/unittests/test-gimple.c
create mode 100644 gcc/unittests/test-hash-map.c
create mode 100644 gcc/unittests/test-hash-set.c
create mode 100644 gcc/unittests/test-locations.c
create mode 100644 gcc/unittests/test-rtl.c
create mode 100644 gcc/unittests/test-tree.c
create mode 100644 gcc/unittests/test-vec.c
create mode 100644 gcc/unittests/test-wide-int.c
create mode 100644 gcc/unittests/unittests-frontend.c
create mode 100644 gcc/unittests/unittests-main.c