This is the mail archive of the gcc-patches@gcc.gnu.org mailing list for the GCC project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [RFC, v2] Test coverage for --param boundary values


On 07/15/2016 09:22 AM, Thomas Schwinge wrote:
> Hi!
> 
> On Fri, 8 Jul 2016 14:47:46 +0200, Martin Liška <mliska@suse.cz> wrote:
>> From f84ce7be4a998089541fb4512e19f54a4ec25cf6 Mon Sep 17 00:00:00 2001
>> From: marxin <mliska@suse.cz>
>> Date: Fri, 8 Jul 2016 10:59:24 +0200
>> Subject: [PATCH] Add tests that test boundary values of params
> 
> This became r238249.  Yay for pushing testing boundaries; next is some
> fuzzy testing for GCC?  ;-D
> 
>> gcc/ChangeLog:
>>
>> 2016-07-08  Martin Liska  <mliska@suse.cz>
>>
>> 	* Makefile.in: Append rule for params-options.h.
>> 	* params-options.h: New file.
>>
>> gcc/testsuite/ChangeLog:
>>
>> 2016-07-08  Martin Liska  <mliska@suse.cz>
>>
>> 	* gcc.dg/params/blocksort-part.c: New test.
>> 	* gcc.dg/params/params.exp: New file.
> 
> :-/ (GNU-style ChangeLogs are just so useful... not.)
> 
>> --- /dev/null
>> +++ b/gcc/testsuite/gcc.dg/params/blocksort-part.c
>> @@ -0,0 +1,706 @@
>> +
>> +/*-------------------------------------------------------------*/
>> +/*--- Block sorting machinery                               ---*/
>> +/*---                                           blocksort.c ---*/
>> +/*-------------------------------------------------------------*/
>> +
>> +/* ------------------------------------------------------------------
>> +   This file is part of bzip2/libbzip2, a program and library for
>> +   lossless, block-sorting data compression.
>> +
>> +   bzip2/libbzip2 version 1.0.6 of 6 September 2010
>> +   Copyright (C) 1996-2010 Julian Seward <jseward@bzip.org>
>> +
>> +   Please read the WARNING, DISCLAIMER and PATENTS sections in the 
>> +   README file.
>> +
>> +   This program is released under the terms of the license contained
>> +   in the file LICENSE.
>> +[...]
> 

Hi Thomas.

> Are there any issues with including that file without the
> accompanying/referenced README and LICENSE files?

Sure, I'll include these files.

> 
>> --- /dev/null
>> +++ b/gcc/testsuite/gcc.dg/params/params.exp
>> @@ -0,0 +1,64 @@
>> +[...]
>> +# GCC testsuite that uses the `dg.exp' driver.
>> +
>> +# Load support procs.
>> +load_lib gcc-dg.exp
>> +
>> +# Initialize `dg'.
>> +dg-init
>> +
>> +proc param_run_test { param_name param_value } {
>> +    global srcdir
>> +    global subdir
>> +
>> +    dg-runtest $srcdir/$subdir/blocksort-part.c "" "-O3 --param $param_name=$param_value"
>> +}
>> +
>> +set fd [open "$objdir/../../params.options" r]
> 
> (I do understand what you're doing there, but) is it kosher to refer to a
> file from GCC's build tree inside the test suite?  I thought the idea was
> that you could run the testsuite without the build tree being available
> -- as much that's possible.  As far as I remember there are a few
> exception to this rule already, so maybe adding one more is not much of a
> problem.  ;-) (I do know that for our internal testing we have a list of
> files that need to be preserved from the GCC build directory for later
> testing without the build directory being available; so I'll add this
> file to the list; Joseph CCed in case he has some additional comments
> after returning from his vacations.)
> 
> <http://news.gmane.org/find-root.php?message_id=%3C201607141327.u6EDRKGV028717%40toolchainslave5.caveonetworks.com%3E>
> shows one instance of the problem, that I could quickly find:
> 
>     [...]
>     ERROR: tcl error sourcing /home/jenkins/workspace/BuildToolchainThunder_elf_test_upstream/toolchain/src/gcc/testsuite/gcc.dg/params/params.exp.
>     ERROR: couldn't open "/home/jenkins/workspace/BuildToolchainThunder_elf_test_upstream/toolchain/testresults/aarch64-thunderx-elf/../../params.options": no such file or directory
>     [...]

You are right, I was inspired by what we do for GCC plugins in:
gcc/testsuite/lib/plugin-support.exp

where we have following comment:
    # Note that the plugin test support currently only works when the GCC
    # build tree is available. (We make sure that is the case in plugin.exp.)
    # Once we have figured out how/where to package/install GCC header files
    # for general plugin support, we should modify the following include paths
    # accordingly.

Well, I can imaging a guard which will test whether "$objdir/../../params.options" file exits,
and if so, then the tests are executed? Is it acceptable approach?

> 
>> +set text [read $fd]
>> +close $fd
>> +
>> +# Main loop.
>> +foreach params [split $text "\n"] {
>> +    set parts [split $params "="]
>> +    set name [string trim [lindex $parts 0] '"']
>> +    set values [split [lindex $parts 1] ","]
>> +    if { [llength $values] == 3 } {
>> +	set default [lindex $values 0]
>> +	set min [lindex $values 1]
>> +	set max [lindex $values 2]
>> +	set int_max "INT_MAX"
>> +
>> +	if { $min != -1 } {
>> +	    param_run_test $name $min
>> +	}
>> +	if { $max != $min && $max > 0 && $max != $int_max } {
>> +	    param_run_test $name $max
>> +	}
>> +    }
>> +    if { [llength $values] == 5 } {
>> +	foreach v $values {
>> +	    param_run_test $name $v
>> +	}
>> +    }
>> +}
> 
> This produces ambiguous test result lines:
> 
>     Running /scratch/tschwing/nvidiak20-2/gcc/trunk-light/source-gcc/gcc/testsuite/gcc.dg/params/params.exp ...
>     PASS: gcc.dg/params/blocksort-part.c (test for excess errors)
>     PASS: gcc.dg/params/blocksort-part.c (test for excess errors)
>     PASS: gcc.dg/params/blocksort-part.c (test for excess errors)
>     PASS: gcc.dg/params/blocksort-part.c (test for excess errors)
>     [many more]
> 
> OK to fix this as follows:
> 
>     PASS: gcc.dg/params/blocksort-part.c -O3 --param predictable-branch-outcome=0 (test for excess errors)
>     PASS: gcc.dg/params/blocksort-part.c -O3 --param predictable-branch-outcome=50 (test for excess errors)
>     PASS: gcc.dg/params/blocksort-part.c -O3 --param inline-min-speedup=0 (test for excess errors)
>     PASS: gcc.dg/params/blocksort-part.c -O3 --param max-inline-insns-single=0 (test for excess errors)
>     [...]
> 
> commit ae350ce9af062d7d2c05225280596b1bc982728b
> Author: Thomas Schwinge <thomas@codesourcery.com>
> Date:   Fri Jul 15 09:13:19 2016 +0200
> 
>     Disambiguate gcc.dg/params/params.exp test results
>     
>     	gcc/testsuite/
>     	* gcc.dg/params/params.exp (param_run_test): Swap flags and
>     	default-extra-flags arguments when calling dg-runtest.
> ---
>  gcc/testsuite/gcc.dg/params/params.exp |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git gcc/testsuite/gcc.dg/params/params.exp gcc/testsuite/gcc.dg/params/params.exp
> index f2f397e..dd82a5d 100644
> --- gcc/testsuite/gcc.dg/params/params.exp
> +++ gcc/testsuite/gcc.dg/params/params.exp
> @@ -28,7 +28,7 @@ proc param_run_test { param_name param_value } {
>      global srcdir
>      global subdir
>  
> -    dg-runtest $srcdir/$subdir/blocksort-part.c "" "-O3 --param $param_name=$param_value"
> +    dg-runtest $srcdir/$subdir/blocksort-part.c "-O3 --param $param_name=$param_value" ""
>  }
>  
>  set fd [open "$objdir/../../params.options" r]
> 
> 
> Grüße
>  Thomas
> 

Thanks for the patch, that's probably the same problem that Andreas spotted.
I'm sending candidate patch.

Martin

>From d5e3c5cb20306352af7f45cd827669b7ff51a709 Mon Sep 17 00:00:00 2001
From: marxin <mliska@suse.cz>
Date: Mon, 18 Jul 2016 09:50:12 +0200
Subject: [PATCH] Fix params.exp test-cases

gcc/testsuite/ChangeLog:

2016-07-18  Thomas Schwinge  <thomas@codesourcery.com>
	Martin Liska  <mliska@suse.cz>

	* gcc.dg/params/LICENSE: Add missing LICENSE that is mentioned
	in a source file.
	* gcc.dg/params/README: Same for the README.
	* gcc.dg/params/params.exp (param_run_test): Swap flags and
	default-extra-flags arguments when calling dg-runtest.
---
 gcc/testsuite/gcc.dg/params/LICENSE    |  42 +++++++
 gcc/testsuite/gcc.dg/params/README     | 215 +++++++++++++++++++++++++++++++++
 gcc/testsuite/gcc.dg/params/params.exp |   2 +-
 3 files changed, 258 insertions(+), 1 deletion(-)
 create mode 100644 gcc/testsuite/gcc.dg/params/LICENSE
 create mode 100644 gcc/testsuite/gcc.dg/params/README

diff --git a/gcc/testsuite/gcc.dg/params/LICENSE b/gcc/testsuite/gcc.dg/params/LICENSE
new file mode 100644
index 0000000..cc61417
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/params/LICENSE
@@ -0,0 +1,42 @@
+
+--------------------------------------------------------------------------
+
+This program, "bzip2", the associated library "libbzip2", and all
+documentation, are copyright (C) 1996-2010 Julian R Seward.  All
+rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+
+2. The origin of this software must not be misrepresented; you must 
+   not claim that you wrote the original software.  If you use this 
+   software in a product, an acknowledgment in the product 
+   documentation would be appreciated but is not required.
+
+3. Altered source versions must be plainly marked as such, and must
+   not be misrepresented as being the original software.
+
+4. The name of the author may not be used to endorse or promote 
+   products derived from this software without specific prior written 
+   permission.
+
+THIS SOFTWARE IS PROVIDED BY THE AUTHOR ``AS IS'' AND ANY EXPRESS
+OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY
+DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE
+GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
+WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
+NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
+SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+
+Julian Seward, jseward@bzip.org
+bzip2/libbzip2 version 1.0.6 of 6 September 2010
+
+--------------------------------------------------------------------------
diff --git a/gcc/testsuite/gcc.dg/params/README b/gcc/testsuite/gcc.dg/params/README
new file mode 100644
index 0000000..9fb0f63
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/params/README
@@ -0,0 +1,215 @@
+
+This is the README for bzip2/libzip2.
+This version is fully compatible with the previous public releases.
+
+------------------------------------------------------------------
+This file is part of bzip2/libbzip2, a program and library for
+lossless, block-sorting data compression.
+
+bzip2/libbzip2 version 1.0.6 of 6 September 2010
+Copyright (C) 1996-2010 Julian Seward <jseward@bzip.org>
+
+Please read the WARNING, DISCLAIMER and PATENTS sections in this file.
+
+This program is released under the terms of the license contained
+in the file LICENSE.
+------------------------------------------------------------------
+
+Complete documentation is available in Postscript form (manual.ps),
+PDF (manual.pdf) or html (manual.html).  A plain-text version of the
+manual page is available as bzip2.txt.
+
+
+HOW TO BUILD -- UNIX
+
+Type 'make'.  This builds the library libbz2.a and then the programs
+bzip2 and bzip2recover.  Six self-tests are run.  If the self-tests
+complete ok, carry on to installation:
+
+To install in /usr/local/bin, /usr/local/lib, /usr/local/man and
+/usr/local/include, type
+
+   make install
+
+To install somewhere else, eg, /xxx/yyy/{bin,lib,man,include}, type
+
+   make install PREFIX=/xxx/yyy
+
+If you are (justifiably) paranoid and want to see what 'make install'
+is going to do, you can first do
+
+   make -n install                      or
+   make -n install PREFIX=/xxx/yyy      respectively.
+
+The -n instructs make to show the commands it would execute, but not
+actually execute them.
+
+
+HOW TO BUILD -- UNIX, shared library libbz2.so.
+
+Do 'make -f Makefile-libbz2_so'.  This Makefile seems to work for
+Linux-ELF (RedHat 7.2 on an x86 box), with gcc.  I make no claims
+that it works for any other platform, though I suspect it probably
+will work for most platforms employing both ELF and gcc.
+
+bzip2-shared, a client of the shared library, is also built, but not
+self-tested.  So I suggest you also build using the normal Makefile,
+since that conducts a self-test.  A second reason to prefer the
+version statically linked to the library is that, on x86 platforms,
+building shared objects makes a valuable register (%ebx) unavailable
+to gcc, resulting in a slowdown of 10%-20%, at least for bzip2.
+
+Important note for people upgrading .so's from 0.9.0/0.9.5 to version
+1.0.X.  All the functions in the library have been renamed, from (eg)
+bzCompress to BZ2_bzCompress, to avoid namespace pollution.
+Unfortunately this means that the libbz2.so created by
+Makefile-libbz2_so will not work with any program which used an older
+version of the library.  I do encourage library clients to make the
+effort to upgrade to use version 1.0, since it is both faster and more
+robust than previous versions.
+
+
+HOW TO BUILD -- Windows 95, NT, DOS, Mac, etc.
+
+It's difficult for me to support compilation on all these platforms.
+My approach is to collect binaries for these platforms, and put them
+on the master web site (http://www.bzip.org).  Look there.  However
+(FWIW), bzip2-1.0.X is very standard ANSI C and should compile
+unmodified with MS Visual C.  If you have difficulties building, you
+might want to read README.COMPILATION.PROBLEMS.
+
+At least using MS Visual C++ 6, you can build from the unmodified
+sources by issuing, in a command shell: 
+
+   nmake -f makefile.msc
+
+(you may need to first run the MSVC-provided script VCVARS32.BAT
+ so as to set up paths to the MSVC tools correctly).
+
+
+VALIDATION
+
+Correct operation, in the sense that a compressed file can always be
+decompressed to reproduce the original, is obviously of paramount
+importance.  To validate bzip2, I used a modified version of Mark
+Nelson's churn program.  Churn is an automated test driver which
+recursively traverses a directory structure, using bzip2 to compress
+and then decompress each file it encounters, and checking that the
+decompressed data is the same as the original.
+
+
+
+Please read and be aware of the following:
+
+WARNING:
+
+   This program and library (attempts to) compress data by 
+   performing several non-trivial transformations on it.  
+   Unless you are 100% familiar with *all* the algorithms 
+   contained herein, and with the consequences of modifying them, 
+   you should NOT meddle with the compression or decompression 
+   machinery.  Incorrect changes can and very likely *will* 
+   lead to disastrous loss of data.
+
+
+DISCLAIMER:
+
+   I TAKE NO RESPONSIBILITY FOR ANY LOSS OF DATA ARISING FROM THE
+   USE OF THIS PROGRAM/LIBRARY, HOWSOEVER CAUSED.
+
+   Every compression of a file implies an assumption that the
+   compressed file can be decompressed to reproduce the original.
+   Great efforts in design, coding and testing have been made to
+   ensure that this program works correctly.  However, the complexity
+   of the algorithms, and, in particular, the presence of various
+   special cases in the code which occur with very low but non-zero
+   probability make it impossible to rule out the possibility of bugs
+   remaining in the program.  DO NOT COMPRESS ANY DATA WITH THIS
+   PROGRAM UNLESS YOU ARE PREPARED TO ACCEPT THE POSSIBILITY, HOWEVER
+   SMALL, THAT THE DATA WILL NOT BE RECOVERABLE.
+
+   That is not to say this program is inherently unreliable.  
+   Indeed, I very much hope the opposite is true.  bzip2/libbzip2 
+   has been carefully constructed and extensively tested.
+
+
+PATENTS:
+
+   To the best of my knowledge, bzip2/libbzip2 does not use any 
+   patented algorithms.  However, I do not have the resources 
+   to carry out a patent search.  Therefore I cannot give any 
+   guarantee of the above statement.
+
+
+
+WHAT'S NEW IN 0.9.0 (as compared to 0.1pl2) ?
+
+   * Approx 10% faster compression, 30% faster decompression
+   * -t (test mode) is a lot quicker
+   * Can decompress concatenated compressed files
+   * Programming interface, so programs can directly read/write .bz2 files
+   * Less restrictive (BSD-style) licensing
+   * Flag handling more compatible with GNU gzip
+   * Much more documentation, i.e., a proper user manual
+   * Hopefully, improved portability (at least of the library)
+
+WHAT'S NEW IN 0.9.5 ?
+
+   * Compression speed is much less sensitive to the input
+     data than in previous versions.  Specifically, the very
+     slow performance caused by repetitive data is fixed.
+   * Many small improvements in file and flag handling.
+   * A Y2K statement.
+
+WHAT'S NEW IN 1.0.0 ?
+
+   See the CHANGES file.
+
+WHAT'S NEW IN 1.0.2 ?
+
+   See the CHANGES file.
+
+WHAT'S NEW IN 1.0.3 ?
+
+   See the CHANGES file.
+
+WHAT'S NEW IN 1.0.4 ?
+
+   See the CHANGES file.
+
+WHAT'S NEW IN 1.0.5 ?
+
+   See the CHANGES file.
+
+WHAT'S NEW IN 1.0.6 ?
+
+   See the CHANGES file.
+
+
+I hope you find bzip2 useful.  Feel free to contact me at
+   jseward@bzip.org
+if you have any suggestions or queries.  Many people mailed me with
+comments, suggestions and patches after the releases of bzip-0.15,
+bzip-0.21, and bzip2 versions 0.1pl2, 0.9.0, 0.9.5, 1.0.0, 1.0.1,
+1.0.2 and 1.0.3, and the changes in bzip2 are largely a result of this
+feedback.  I thank you for your comments.
+
+bzip2's "home" is http://www.bzip.org/
+
+Julian Seward
+jseward@bzip.org
+Cambridge, UK.
+
+18     July 1996 (version 0.15)
+25   August 1996 (version 0.21)
+ 7   August 1997 (bzip2, version 0.1)
+29   August 1997 (bzip2, version 0.1pl2)
+23   August 1998 (bzip2, version 0.9.0)
+ 8     June 1999 (bzip2, version 0.9.5)
+ 4     Sept 1999 (bzip2, version 0.9.5d)
+ 5      May 2000 (bzip2, version 1.0pre8)
+30 December 2001 (bzip2, version 1.0.2pre1)
+15 February 2005 (bzip2, version 1.0.3)
+20 December 2006 (bzip2, version 1.0.4)
+10 December 2007 (bzip2, version 1.0.5)
+ 6     Sept 2010 (bzip2, version 1.0.6)
diff --git a/gcc/testsuite/gcc.dg/params/params.exp b/gcc/testsuite/gcc.dg/params/params.exp
index f2f397e..dd82a5d 100644
--- a/gcc/testsuite/gcc.dg/params/params.exp
+++ b/gcc/testsuite/gcc.dg/params/params.exp
@@ -28,7 +28,7 @@ proc param_run_test { param_name param_value } {
     global srcdir
     global subdir
 
-    dg-runtest $srcdir/$subdir/blocksort-part.c "" "-O3 --param $param_name=$param_value"
+    dg-runtest $srcdir/$subdir/blocksort-part.c "-O3 --param $param_name=$param_value" ""
 }
 
 set fd [open "$objdir/../../params.options" r]
-- 
2.8.4


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]