Re: Asan/Tsan Unit/Regression testing (was [asan] Emit GIMPLE direclty, small cleanups)

[Jakub Jelinek <jakub at redhat dot com>]

On Wed, Nov 07, 2012 at 07:02:04AM -0800, Kostya Serebryany wrote:
> 1. LLVM unittests (Text file with LLVM IR and the expected result of
> the transformations).
> Example:  http://llvm.org/viewvc/llvm-project/llvm/trunk/test/Instrumentation/AddressSanitizer/do-not-touch-threadlocal.ll?revision=145092&view=markup
> I am not sure if anything similar is possible with GCC.

We don't have a GIMPLE FE (yet), so we need to feed in C or C++ or Fortran
etc. as source (which isn't that hard though), but can scan gimple and/or
rtl dumps and/or assembly for required transformations easily.

> 2. Large Gtest-based unittest. This is a set of c++ files that should
> be built with the asan switch, depends on gtest
> (http://code.google.com/p/googletest/).
> http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/asan/tests/asan_test.cc?revision=166104&view=markup
> This should be easy to port to GCC, but it requires gtest.

I don't think we want to depend on gtest, if the tests only use a small
subset of that, it would be better to just use some header with macros
compatible with that for assertions and the like.

> 3. Full output tests (a .cc file should be build with asan switch,
> executable should be run and the stderr is compared with the expected
> output)
> Example: http://llvm.org/viewvc/llvm-project/compiler-rt/trunk/lib/asan/lit_tests/stack-overflow.cc?revision=165391&view=markup
> The can be ported to GCC, but the uses of FileCheck
> (http://llvm.org/docs/CommandGuide/FileCheck.html) will need to be
> replaced with GCC's analog.
> We should probably start with these tests.

Dejagnu in GCC has

! { dg-do run }
! { dg-options "-fbounds-check" }
! { dg-shouldfail "Duplicate value 2 in ORDER argument to RESHAPE intrinsic" }
program main
  implicit none
  integer(kind=1), dimension(6) :: source1 = (/ 1, 2, 3, 4, 5, 6 /)
  integer, dimension(2) :: shape1 = (/ 2, 3/)
  integer(kind=1), dimension(2) :: pad1 = (/ 0, 0/)
  character(len=200) :: l1, l2
  integer :: i1, i2

  l1 = "2 2"
  read(unit=l1,fmt=*) i1, i2
  write (unit=l2,fmt=*) reshape(source1, shape1, pad1, (/i1, i2/)) ! Invalid
end program main
! { dg-output "Fortran runtime error: Duplicate value 2 in ORDER argument to RESHAPE intrinsic" }

style markings, dg-shouldfail says that the program is expected to fail
rather than pass (if it aborts), and dg-output (perhaps multiple) can
contain regexps to match against stderr + stdout joined.  Haven't looked
at the asan tests yet, do you expect just one ASAN abort per test, or do you
say somehow catch the failure exception, longjmp to some safe point and
continue with another crash?  I mean something similar to glibc
-D_FORTIFY_SOURCE tests,
http://sourceware.org/git/?p=glibc.git;a=blob;f=debug/tst-chk1.c;h=7ffc9fec9084a1f536c562d4dc3861055b7b5788;hb=HEAD
?

	Jakub