Re: Meta-expectations when testing

[Hugh Sasse Staff Elec Eng <hgs at dmu dot ac dot uk>]

On Tue, 25 Feb 2003, Janis Johnson wrote:

> On Tue, Feb 25, 2003 at 10:41:57AM +0000, Hugh Sasse Staff Elec Eng wrote:
> > On Mon, 24 Feb 2003, Janis Johnson wrote:
> > > On Mon, Feb 24, 2003 at 04:52:44PM +0000, Hugh Sasse Staff Elec Eng wrote:
> > > > Should I "expect the unexpected" failures or not?  There is some
        [...]
> > > The GCC build status lists have links to archived test results for many
> > > targets, so you can often find a similar target with test results to
> > > compare with your own; see http://gcc.gnu.org/gcc-3.2/buildstat.html.
> >
> > I found the same target and it said the build was successful. Other
        [...]
>
> Follow the links for test results.  Many people report successful builds
> without even running the test suites, let alone submitting the test

This was the case here. Thanks for pointing that out.

I'd like to propose the following patch to the test.html:
Its purposes: to point that out and to put a reference in there to
the list of results, which doesn't seem to be on the test page.

--- test.html.orig      2003-02-25 18:49:09.360007000 +0000
+++ test.html   2003-02-25 18:55:30.920007000 +0000
@@ -178,6 +178,12 @@
 over whether or not a test is expected to fail.  We expect to fix this
 problem in future releases.

+   <p>You may find it useful to compare your results with those from
+similar builds listed at <a
+href="http://gcc.gnu.org/buildstat.html";>http://gcc.gnu.org/buildstat.html</a>,
+but be aware that sometimes a build listed as successful has no test
+results associated with it.
+
 <h3 class="section"><a name="TOC3"></a>Submitting test results</h3>

    <p>If you want to report the results to the GCC project, use the



I just tried to submit my results and it failed to work:
I had no comments to make as such but to keep the command the same I
did
touch no_comment
../gcc-3.2.1/contrib/test_summary -p no_comment -m gcc-testresults at gcc dot gnu dot org |sh
tail: cannot open input

        [...]
>
> Ideally, in released versions of GCC all tests that are known to fail
> for a particular target would be marked as expected failures, but that
> requires a lot of work and seldom happens.

Yes, I understand that.
>
> Janis
>
        Thank you,
        Hugh