Re: Meta-expectations when testing

[Janis Johnson <janis187 at us dot ibm dot com>]

On Tue, Feb 25, 2003 at 10:41:57AM +0000, Hugh Sasse Staff Elec Eng wrote:
> On Mon, 24 Feb 2003, Janis Johnson wrote:
> > On Mon, Feb 24, 2003 at 04:52:44PM +0000, Hugh Sasse Staff Elec Eng wrote:
> > > Should I "expect the unexpected" failures or not?  There is some
> > > evidence in the documentation on the testing pages that this testing
> > > code is not as up-to-date as they maintainers would wish, (given the
> > > usual constraints on volunteer based softare development), so this
> > > may not be a daft a question as it looks :-) I hope!
> >
> > The GCC build status lists have links to archived test results for many
> > targets, so you can often find a similar target with test results to
> > compare with your own; see http://gcc.gnu.org/gcc-3.2/buildstat.html.
> 
> I found the same target and it said the build was successful. Other
> similar cases (3.2.2) referred to errors.  Therefore I concluded
> that I should get no errors.  Hence my question as to how to
> determine if the discrepancy is significant.  I can see that this is
> not a simple question because the significance may well depend on
> what software one is building with GCC.

Follow the links for test results.  Many people report successful builds
without even running the test suites, let alone submitting the test
results, and having some unexpected test failures wouldn't cause someone
to report that the build had problems.  If there are no test results for
your GCC version and target, look at results for the same version but
slightly different targets.

Ideally, in released versions of GCC all tests that are known to fail
for a particular target would be marked as expected failures, but that
requires a lot of work and seldom happens.

Janis