RFC: run tests on installed compiler

[Benjamin Kosnik]

I am going to rename this discussion to preserve sanity.

> This patch makes it possible to run the V3 testsuite on an installed
> compiler, which us allows us to validate the bits we actually ship.

Sounds great!

> The V3 testsuite was (long ago) converted to DejaGNU from its previous
> format.  Perhaps for this reason, the testsuite was not set up to work
> well with installed compilers.

What did you have in mind? If you can be specific about what kind of
process you are aiming for it would be helpful. So you have an installed
toolchain. What else to you need to test it? What kind of process are
you aiming for? 

There is a rule (make check-script, make check-script-install) and a
crufty script that allow checking of installed libraries, both static
and shared.

Is this the kind of thing you want to do, but better? Sounds cool.

Is the g++ testsuite testable like this? If so, let's do something similar. 

I would like to see a toplevel make rule, called "make check-install"
that does this for the entire toolchain. It's ok if we don't get there
all at once, but I think we should keep that in mind.

> It is wrong for the V3 Makefiles to build anything that will be an
> input to the test process, unless that thing is installed with V3.
> For example, libv3test.a should not be built as part of the ordinary
> build process, because it is (rightly) not installed.  Instead, these
> kinds of things should be built as part of the test process, just as
> we already handle things like testglue.o.

Ummm. Ok. 

> In this patch, I fixed the most important cases of this problem: the
> use of the testsuite_files file, and the use of libv3test.a.  (In the
> case of the former, we now generate the list of tests dynamically; in
> the case of the latter, we build the objects at test-time.)  I did not
> correct the handling of fr.po and de.po; these, too, should be built
> at test time, with the result that LOCALEDIR should just be the
> current directory.  I will work on that in the near future.

My preference is to allow use of testsuite_file, but make it optional,
so that in its absence, the list of files is autogenerated. I believe
this is the preference of Phil, and Mike Stump as well. Perhaps others
have a preference too: if so, this is probably a good time to say so.

The rest I have no problem with. (ie, LOCALEDIR, building v3test.a at test time.)

> 1. Remove check_compile, in favor of providing an option to DejaGNU to
>    just generate assembly files, if people still need this
>    functionality.

Suggestion: come up with the dejagnu magic to do this, and then talk about
removing this rule. If you think this functionality is of more general
use than just libstdc++, and want to push it everywhere: cool. Keep the
rule though, it's useful.

"People" like me and "Red Hat" need this rule to assess compile time
performance: recent gcc releases have regressed quite noticeably, compile
time. This gives us an easy way to scope things. 

See this for how it's used:
http://people.redhat.com/bkoz/benchmarks/

> 2. Modify check_performance to work without testsuite_files, probably
>    by integrating that functionality with DejaGNU.

Have you ever run this rule? BTW it uses testsuite_files_performance.

Try running 'make check-performance.' See how it eats machine resources.

Keeping this as a separate rule is a firm requirement: there's no way we
can inflict this on everybody.

Interestingly, this and the ulimits stuff is a direct response to Zack
complaining about this very issue, ages ago.

So. 

There you go.

In general, I'd like to see the focus on making installation testable.
Let's start there! Wholesale conversion of specialized libstdc++ make
check-* rules to conform to dejagnu's scary acid-trip can be undertaken
later, huh? 

-benjamin