Re: improving testing scheme (was Re: ACATS and GCC testsuite)

[kenner at vlsi1 dot ultra dot nyu dot edu (Richard Kenner)]

    If this is really impossible for you to do right, make a little script
    that does the necessary changes. You are making very heavy weather of
    a simple matter.

But remember that those "scripts" will have to be site-dependent, so their
creation will add yet more complexity to the configuration process.  The
issue isn't what "possible" or "impossible", but which of the two 
procedures is the most *reliable*.

The bottom line to me is that I don't see the major advantage in changing to
testing an installed compiler over what we do now.

Right now, "make check" is run from the build directory and tests the
compiler that was just built into that directory.  This is fairly
"idiot-proof": if the compiler wasn't built in that directory, it won't work
and there is no other place that "make check" looks for the compiler.  So if
somebody says they've run "make check", we know they've done it on the
compiler they in fact made their change on.

The single disadvantage of doing it this way is that we don't test "make
install".  However, I don't see this as a major problem: we don't change it
very often.  Moreover, to verify you've properly installed the compiler, you
don't need to run thousands of C programs through it: a handful are fine.  So
I think the testing of "make install" is of a different type, more like
sanity checking.

The risk, in my opinion, of changing "make check" to use the compiler in
PATH, rather than in the build directory, is that a lot of people are going
to accidentally run the check on the wrong compiler.  The main reason for
this will simply be "inertia": they are use to "make check" doing something
else for quite a while.  So it's easy to forget either "make install" or
changing the path, or both.  Having the maintainer-mode installation
prompt the user for configuration information and make new rules to
accomodate this is possible, but adds more complexity.

I think the risk of having "make check" run on the wrong compiler is far
larger than the risk of problems in "make install" that don't get detected.

I do think it worthwhile to add a new target "make check-install" that does a
(much shorter) check on the compiler in PATH to make sure everything is
installed.

I've also heard a proposal to rearrange things so the compiler is built
directly into a directory structure identical to that created by "make
install", so the latter is simply a bulk copy.  This will also solve the
problem without creating new ones.

Can you explain more about why you think the risk of procedural error here is
small enough that the small gain of testing "make install" overrides it?