Re: Meta-expectations when testing

[Hugh Sasse Staff Elec Eng <hgs at dmu dot ac dot uk>]

On Mon, 24 Feb 2003, Janis Johnson wrote:

> On Mon, Feb 24, 2003 at 04:52:44PM +0000, Hugh Sasse Staff Elec Eng wrote:
> >
> > As far as I can gather from the listing on the GCC tests page
> > GCC-3.2.1 should build OK on Solaris2.9, and pass the tests.
> > However on my system I get the following results at the end of
        [...]
> > # of expected passes            410
> > # of unexpected failures        1
> > # of unexpected successes       10
> > # of expected failures          16
>
> Almost every target has some unexpected failures.

OK, than how is one to tell which are significant?
>
> > gmake[4]: *** [check-DEJAGNU] Error 1
        [...]
> > gmake: *** [check] Error 2
>
> This is normal when there are some tests that fail.  Use "make -k check"
> so that make will ignore errors.

OK, I forgot the -k, but I suspect that some errors should not be
ignored.
>
> > Should I "expect the unexpected" failures or not?  There is some
> > evidence in the documentation on the testing pages that this testing
> > code is not as up-to-date as they maintainers would wish, (given the
> > usual constraints on volunteer based softare development), so this
> > may not be a daft a question as it looks :-) I hope!
>
> The GCC build status lists have links to archived test results for many
> targets, so you can often find a similar target with test results to
> compare with your own; see http://gcc.gnu.org/gcc-3.2/buildstat.html.

I found the same target and it said the build was successful. Other
similar cases (3.2.2) referred to errors.  Therefore I concluded
that I should get no errors.  Hence my question as to how to
determine if the discrepancy is significant.  I can see that this is
not a simple question because the significance may well depend on
what software one is building with GCC.
>
> Janis
>
        Thank you,
        Hugh