Re: btest-gcc.sh patch ping

[Hans-Peter Nilsson <hp at bitrange dot com>]

On Mon, 3 Dec 2007, Geoff Keating wrote:
> Part of the integrity of the result is that the same things are tested in
> every run of the script on a particular platform.  The script was specifically
> intended to not be configurable in the manner you want, because that would
> allow for the possibility that a misconfigured tester might miss some tests
> that ought to be run.

The same original set of tests are still run on all platforms.
If there are more that "ought to be run", I suggest you add
them.  Regarding missing tests, you must mean some other patch.
The way $BTEST_GCC_EXTRA_TESTLOGS is added in the patch, just as
with the "static" logs, it will be an error if the file(s) do
not exist and contains "PASS:".  ("More stable" than the
optional test logs added on an "if exists" basis above the
patched area, for example.)

> > > If there's a log missing, and you think the underlying functionality is
> > > stable
> > > and well supported by GCC contributors on all platforms where it is
> > > enabled,
> > > propose a patch to add it.
> >
> > The fortran tests are stable where it works, or so it seems.
> > Whether it works is port-specific (build issue on *-elf being
> > covered in this thread), and the patch above was a way to add
> > the functionality per-platform.  It would also allow e.g.
> > tracking gas or ld results in a combined tree.
>
> The patch does not accomplish this.  It instead adds the functionality
> *per-tester*.  That's different to per-platform.

It does too, but better: rather than the static list you seem to
imply, it's enables the *tester* control over which optional
testlogs apply to which platform.

> > I just need a regtest-script I can point to, such that people can see how I
> > build and regtest and possibly repeat it.
>
> I would not go look at the scripts if I was trying to reproduce the results of
> my tester.  I would instead look at the build log which shows the exact
> 'configure' and 'make' commands used, with no need to do shell variable
> substitution in my head.

By "repeat" I did not mean a specific run but to *reusing the
script for their own setup*.  The build_log just as before
contains all details about the specific run, including whatever
value BTEST_GCC_EXTRA_TESTLOGS had at the time.

> > Your btest-gcc.sh mostly fits, but not covering all gcc-related tests that
> > run by "make check" is really an issue.
>
> The patch does not implement this, either, but it might be a good idea.

Yes it does too again: the tester will have to know beforehand
which logs apply, but I think that makes more sense than using
"find" and adding all testsuite logs willy-nilly.

> The reason the script doesn't cover all tests is because some parts of the
> testsuite were too unreliable to be used in an automated fashion.  That was
> some time ago and it may be that today it should simply look at all the
> testsuite.  Or, not.  If you have some data that points either way it would be
> very helpful.

I wouldn't do it by default, but I agree e.g. a --find-all-logs
option might be useful.  I don't have a use for it myself, I
just meant it as an example of what the patch enables.

> > I guess I could solve that by adding a variant of that script in contrib/
> > though I do prefer improving the existing one.
> Why?
>
> I can't imagine you're saving an enormous amount of time in maintenance.  Nor
> is this script more conveniently located than a script you maintain in your
> own tree.  You could put your script on a web server and point people at that.

I could say your arguments speak more against
contrib/regression/btest-gcc.sh being there, but I don't agree
with your arguments in either case.

On the contrary, I think there's a use for a script dealing with
the housekeeping of an automatic regression tester, and with
some optional aspects enabling different setups.  I just mistook
your script for being it.  I also think the place for such a
script is naturally in contrib/ - is really not the convenience
of having the script there, both for maintenance and sharing
obvious to you?  If you mean something else, speak in clear text.

I propose adding a "less stringent" version as
contrib/build-and-regtest-gcc.sh.

If someone already has their own, different, script for
autotesting they are willing to contribute and is open to
changes while maintaining the default functionality, a script
which is no less than btest-gcc.sh (e.g. handles adding new
tests and pruning old ones), I don't mind switching my
autotester over.

brgds, H-P