Re: Using bugzilla to track XFAILs

["Joseph S. Myers" <jsm at polyomino dot org dot uk>]

On Fri, 13 Feb 2004, Giovanni Bajo wrote:

> it looks like we currently don't have any way to track regressions which are
> XFAILed in the testsuite. Those regressions are not less important than the

If we ran the current testsuite against old releases as well as against
the current release branch, as part of preparing for a release (and then
compared the results), they would show up as XPASS with the old compiler.

(This is in addition to recording XFAILs as known bugs in Bugzilla, which
I think is a good idea - and for all XFAILs, not just those that are
regressions.)

See <http://gcc.gnu.org/gcc-2.95/testing.html> for how such testing with
the same testsuite for an old release and 2.95 branch was done for the
2.95.3 release.  Revision 1.102 of gcc-2.95/regress.html has the
instructions that were used for regression testing for the 2.95 release:

======================================================================
<p>One of the critical steps in making releases is regression testing.  For
gcc-2.95 we want to perform regression tests against egcs-1.1.2.

<p>Since it is impossible to test every platform, we have selected a list
of popular platforms for which gcc-2.95 should have no regressions 
relative
to egcs-1.1.2 for gcc, g++, g77, objc, libstdc++ and libio.

We also want to run libf2c and libjava testsuites to verify some basic
functionality.

<p>We encourage testing of platforms not on the official regression test  
list.  However, the "no regressions" policy does not necessarily apply
to targets not on the official regression test list.

<p>Contact <a 
href="mailto:cheewai_yeung@yahoo.com";>cheewai_yeung@yahoo.com</a>
directly if you want to help with the regression testing phase of the gcc-2.95
release cycle.

<h2>How to run the regression tests</h2>

<p>  First, you must get out of the egcs object directory.  So change
     directories into your home directory, /tmp or somewhere else.

<p>  Second, you need an appropriate site.exp.  Here is a template:

<pre>
# snapdir should point to the top-level directory of an unpacked gcc-2.95
# snapshot
set snapdir "/some/place/gcc-19990415"

# set old_release_prefix to the prefix directory of an installed 
egcs-1.1.2.
set old_release_prefix /prefix/for/egcs-1.1.2

set rootme "."
set target_triplet [exec "$snapdir/config.guess"]
set srcdir "$snapdir/gcc/testsuite"
set GCC_UNDER_TEST "$old_release_prefix/bin/gcc"
set GXX_UNDER_TEST "$old_release_prefix/bin/g++"
set G77_UNDER_TEST "$old_release_prefix/bin/g77"
</pre>

<p>Obviously you have to change "snapdir" and "old_release_prefix" to
appropriate values.

<p>Then run the tests with the following commands:

<pre>
  runtest --tool gcc
  runtest --tool g++
  runtest --tool g77
  runtest --tool objc
</pre>


<p>This will create "gcc.sum", "g++.sum", "g77.sum" and objc.sum.  You 
should
send those files to the testing coordinator
<a href="mailto:cheewai_yeung@yahoo.com";>cheewai_yeung@yahoo.com</a>.

<p> We will also perform some tests on the runtime libraries, but there is
no need to run those tests with the egcs-1.1.2 based compiler.  Details for
running and these test results will be added soon.
======================================================================

I don't know about any particular scripts used to compare the test results
(but contrib/compare_tests should do the job); runtime libraries and Ada
should be covered as well; test results best go to gcc-testresults if not
too big; and no doubt these instructions are out of date in various ways;
but the principle that the testsuite should be used to avoid regressions
in releases not just on platforms with automated regression testers still
applies.

(It would also be nice for the release not to have any unexpected FAILs on
the primary release platforms.  That way, all regressions in the testsuite
would show up as XPASSes with the old compiler.  Cf.
<http://gcc.gnu.org/ml/gcc-patches/2003-10/msg02559.html>.)

-- 
Joseph S. Myers
jsm@polyomino.org.uk