Re: [PATCH 00/17] RFC: Addding a unit testing framework to gcc

[David Malcolm <dmalcolm at redhat dot com>]

On Wed, 2015-06-10 at 17:34 +0200, Jakub Jelinek wrote:
> On Wed, Jun 10, 2015 at 11:24:41AM -0400, David Malcolm wrote:
> > I picked the Google Test framework:
> >   http://code.google.com/p/googletest/
> 
> I must say I'm not very excited about using this, it won't integrate
> very well with dejagnu

Why is that a goal?  I've been using DejaGnu's unittesting API for
testing the jit, and it is... suboptimal, to put it mildly.

> whether talking about results (will it provide
> some *.log/*.sum file with FAIL/XFAIL/PASS/XPASS etc. lines?),

It doesn't have an output formatter for the DejaGnu format, but I guess
I could write one.  The gtest standard output format is IMHO superior to
DejaGnu's since it tells you start-of-test/end-of-test on separate
lines, so you can see which test killed things in the event of total
failure, and the per-test timings (which can be disabled if you want to
do diffs).

>  choosing
> what options to use, e.g. global
> RUNTESTFLAGS='--target_board=unix/\{-m32,-m64\}'
> to test everything twice for 32-bit and 64-bit, will that run just
> all unittests twice the same?,

Fair enough; yes, as written, RUNTESTFLAGS is ignored; it will run
everything once.  However we could express such a loop directly in the
"main" of the 

> or possibility to run a subset of tests

$ LD_LIBRARY_PATH=. ./unittests.exe --gtest_filter="*push"
Note: Google Test filter = *push
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from vec_test
[ RUN      ] vec_test.quick_push
[       OK ] vec_test.quick_push (0 ms)
[ RUN      ] vec_test.safe_push
[       OK ] vec_test.safe_push (0 ms)
[----------] 2 tests from vec_test (1 ms total)

[----------] Global test environment tear-down
[==========] 2 tests from 1 test case ran. (4 ms total)
[  PASSED  ] 2 tests.

> etc.
> E.g. for asan.exp testing, I just wrote a gtest emulation using
> dejagnu, see testsuite/g++.dg/asan/dejagnu-gtest.h and
> testsuite/lib/asan-dg.exp

...which doesn't have things like EXPECT_STREQ, or custom comparators,
doesn't appear to support fixtures, type-parameterized tests,
value-parameterized tests, etc, my point being that a unit-testing
framework is a non-trivial amount of code that could do useful things
for us.

> but that was mainly meant for cases where
> many routines are expected to crash the process.

FWIW, if I'm reading testsuite/g++.dg/asan/dejagnu-gtest.h and
testsuite/lib/asan-dg.exp correctly, it looks like you're spawning the
tool once per EXPECT_DEATH instance in the testsuite.  That seems
suboptimal, gtest uses fork without exec
to do it all directly at the point of the test without lots of extra
work, where the parent verifies the death of the child.

>   If in unittests
> you are doing only operations that aren't meant to take everything down or
> if they crash, it is ok if it breaks the whole unit testing,
> then perhaps it can be run as a single process and thus a single dejagnu
> job, and just let the wrapper parse the output and transform it.

I guess, but having used google test lately, DejaGnu feels like a big
step backwards.

> Also, no matter what testsuite framework is used, including any
> headers before #include "config.h" line is a big no-no.

OK.  Can't remember why I did that.


Dave