Re: Meta-expectations when testing

[Hugh Sasse Staff Elec Eng <hgs at dmu dot ac dot uk>]

On Wed, 26 Feb 2003, Janis Johnson wrote:

> On Wed, Feb 26, 2003 at 09:59:40AM +0000, Hugh Sasse Staff Elec Eng wrote:
> > On Tue, 25 Feb 2003, Janis Johnson wrote:
> [snip]
> > >
> > > Follow the links for test results.  Many people report successful builds
> > > without even running the test suites, let alone submitting the test
> >
> > This was the case here. Thanks for pointing that out.
> >
> > I'd like to propose the following patch to the test.html:
> > Its purposes: to point that out and to put a reference in there to
> > the list of results, which doesn't seem to be on the test page.
> >
        [patch elided.
>
> The first sentence in test.html (generated from gcc/doc/install.texi) is:
>
>   Before you install GCC, we encourage you to run the testsuites and to
>   compare your results with results from a similar configuration that have
>   been submitted to the @uref{http://gcc.gnu.org/ml/gcc-testresults/,,
>   gcc-testresults mailing list}.

Yes, but that is on the install page.  When you are testing you will
have the testing page open.  I think that one wants this information
when one runs the tests, and before one has exhauseted all the
possibilities of what has gone wrong in running the tests
[forgetting -k, not haveing the

         TCL_LIBRARY = /usr/local/share/tcl8.0
         DEJAGNULIBS = /usr/local/share/dejagnu

set, and so forth], because when one gets to the bit about submitting
one's test results one wants to submit something sensible.  I could
never remember where I'd seen it, and it was a job finding it again.
[Sometimes the linking in the web feels like "a lot of twisty little
passages, all alike" :-)]

I know one should not repeat information unnecessarily (because of
correcting it later) and HTML has no include facility, but I found
having this information on another page rather inconvenient.

I would be satisfied if, to avoid duplication, there was a link "up"
to install.html#where_the_buildstats_are, or something like that.


> Would appending the following to that be sufficient?
>
>   Some of these archived results are linked from the build status lists
>   at @uref{http://gcc.gnu.org/buildstat.html}, although not everyone who
>   reports a successful build runs the testsuites and submits the results.

Not on the install page, for the reasons above. I really believe
that the stepwise approach on the install page, (download,
configure, build, test, install) should obviate the need to go
backwards to find information.

I don't build GCC very often, and don't consider myself familiar
with the details of the process, and anything that makes it easier
for people in my position will reduce traffic to the list.
>
> (Also, the maintainer of the build status lists sometimes doesn't have
> time to go through the gcc-testresults archives after a release to
> include new reports in the build status list as she ought to, and
> welcomes mail pointing out the URLs of such reports, or patches to the
> status list.)

There seems to be no address for a person on these pages, only for
groups of people {gcc,gnu} at gnu dot org, so I don't know who that is.
>
> To me, it seems clear enough already that a successful build doesn't
> imply that all of the tests passed.

Yes, but the question I posed was about how to interpret the
failures.  The "similar platforms" is an effective answer to this.
>
> >
> > I just tried to submit my results and it failed to work:
> > I had no comments to make as such but to keep the command the same I
> > did
> > touch no_comment
> > ../gcc-3.2.1/contrib/test_summary -p no_comment -m gcc-testresults at gcc dot gnu dot org |sh
> > tail: cannot open input
> >
> >         [...]
>
> The script moves the *.sum and *.log files to new names, so you can only
> run it once unless you move the files back to their original names.  You

Yes,. I re-ran the tests, then ran it once.
OK, I'll fiddle with that some more.

> shouldn't need to supply any options, since the default is to mail to
> the address you specified.  If you direct the output to a file then you

Good point!
> can look it over before running it, and that way it's available to
> modify and run again if there were problems the first time.
>
> Janis
>

        Thank you again,
        Hugh