Re: Meta-expectations when testing

[Janis Johnson <janis187 at us dot ibm dot com>]

On Mon, Feb 24, 2003 at 04:52:44PM +0000, Hugh Sasse Staff Elec Eng wrote:
> 
> As far as I can gather from the listing on the GCC tests page
> GCC-3.2.1 should build OK on Solaris2.9, and pass the tests.
> However on my system I get the following results at the end of
> running make check, and I don't know if the deviations are
> significant or not.
>         [...[
> bstdc++-v3 Summary ===
> 
> # of expected passes            410
> # of unexpected failures        1
> # of unexpected successes       10
> # of expected failures          16

Almost every target has some unexpected failures.

> gmake[4]: *** [check-DEJAGNU] Error 1
> gmake[4]: Leaving directory `/scratch/hgs/gcc-build/sparc-sun-solaris2.9/libstdc++-v3/testsuite'
> gmake[3]: *** [check-am] Error 2
> gmake[3]: Leaving directory `/scratch/hgs/gcc-build/sparc-sun-solaris2.9/libstdc++-v3/testsuite'
> gmake[2]: *** [check-recursive] Error 1
> gmake[2]: Leaving directory `/scratch/hgs/gcc-build/sparc-sun-solaris2.9/libstdc++-v3'
> gmake[1]: *** [check-target-libstdc++-v3] Error 2
> gmake[1]: Leaving directory `/scratch/hgs/gcc-build'
> gmake: *** [check] Error 2

This is normal when there are some tests that fail.  Use "make -k check"
so that make will ignore errors.

> Should I "expect the unexpected" failures or not?  There is some
> evidence in the documentation on the testing pages that this testing
> code is not as up-to-date as they maintainers would wish, (given the
> usual constraints on volunteer based softare development), so this
> may not be a daft a question as it looks :-) I hope!

The GCC build status lists have links to archived test results for many
targets, so you can often find a similar target with test results to
compare with your own; see http://gcc.gnu.org/gcc-3.2/buildstat.html.

Janis