RFC: run tests on installed compiler

[Mark Mitchell]

Benjamin Kosnik wrote:

>>The V3 testsuite was (long ago) converted to DejaGNU from its previous
>>format.  Perhaps for this reason, the testsuite was not set up to work
>>well with installed compilers.
> 
> 
> What did you have in mind? If you can be specific about what kind of
> process you are aiming for it would be helpful. So you have an installed
> toolchain. What else to you need to test it? What kind of process are
> you aiming for? 

My high-level goal is to be able to validate the actual compilers that 
we provide to customers, in the same environment that the user will use 
them.

Technically, what I want to be able to do is the following three-step 
process:

1. Put compilers I want to test in my PATH.  Set LD_LIBRARY_PATH, if I 
need to do that to run programs copmiled with this compiler.  In other 
words, set things up as I would if I were just going to use this 
compiler to build an application.

2. Create a site.exp file that contains only the host, build, and target 
machine triplets.

3. Run:

runtest --tool libstdc++ --srcdir=/path/to/libstdc++-v3/testsuite

I want that command to run the entire V3 testsuite to be run, and 
produce the exact same results that I would get from "make check" in the 
build tree.

This exact procedure already works for (at least) Binutils, GAS, GCC, 
and G++.  (It doesn't quite work for GNU ld at the moment, but I have a 
patch for that which I'll be submitting to binutils.)

> There is a rule (make check-script, make check-script-install) and a
> crufty script that allow checking of installed libraries, both static
> and shared.
> 
> Is this the kind of thing you want to do, but better? Sounds cool.

Yes, that's the idea.  However, I want to be able to use DejaGNU both 
for consistency with the rest of the toolchain, and for its ability to 
handle cross environments.  I want to be able to do this with no trace 
of the build environment around, because I want to prove to myself that 
there's absolutely no accidental pollution coming from that.

> Is the g++ testsuite testable like this? If so, let's do something similar. 

Yes, the G++ testsuite works exactly like this.  The fundamental 
requirement is that the testsuite has to avoid depending on anything 
built during the build process that is not installed.

Of course, as Mike suggested, it can make use of those things 
optionally, but, as Dan and I pointed out, you really want to avoid 
that, because it's likely to eventually result in differences in 
testsuite runs between testing in the build-tree and testing with 
installed toolchains.

(Even automatically reading testsuite_files if it exists is potentially 
a problem in this way, in that it means that the non-testsuite_files 
codepath will probably be rarely used, and therefore more likely to be 
in disrepair.  However, in my opinion, reading testsuite_files if it 
exists is much less risky than, for example, building libv3test.a at 
build time, which will result in duplicative logic to actually build the 
library.)

> I would like to see a toplevel make rule, called "make check-install"
> that does this for the entire toolchain. It's ok if we don't get there
> all at once, but I think we should keep that in mind.

I think it belongs in the existing "contrib/test_installed" instead. 
You can copy that script to your test machine and run it; it's not 
desirable to have any part of the build machinery around.  I suppose we 
could have "make check-install" run "test_installed", if that was useful.

>>In this patch, I fixed the most important cases of this problem: the
>>use of the testsuite_files file, and the use of libv3test.a.  (In the
>>case of the former, we now generate the list of tests dynamically; in
>>the case of the latter, we build the objects at test-time.)  I did not
>>correct the handling of fr.po and de.po; these, too, should be built
>>at test time, with the result that LOCALEDIR should just be the
>>current directory.  I will work on that in the near future.
> 
> My preference is to allow use of testsuite_file, but make it optional,
> so that in its absence, the list of files is autogenerated. I believe
> this is the preference of Phil, and Mike Stump as well. Perhaps others
> have a preference too: if so, this is probably a good time to say so.

I would prefer that testsuite_file not be used, but I will defer to your 
instructions.  Right now, I'm interpreting what you've said as an 
opinion, not a firm direction to change the way it works.  If you want 
me to change it, please tell me do that explicitly.

> The rest I have no problem with. (ie, LOCALEDIR, building v3test.a at test time.)

Excellent.

>>1. Remove check_compile, in favor of providing an option to DejaGNU to
>>   just generate assembly files, if people still need this
>>   functionality.
> 
> 
> Suggestion: come up with the dejagnu magic to do this, and then talk about
> removing this rule. If you think this functionality is of more general
> use than just libstdc++, and want to push it everywhere: cool. Keep the
> rule though, it's useful.

OK.  I think this is easy to do.

>>2. Modify check_performance to work without testsuite_files, probably
>>   by integrating that functionality with DejaGNU.
> 
> 
> Have you ever run this rule? BTW it uses testsuite_files_performance.

Yes, I understand, and by "integrate" I did not mean "run these tests by 
default".  I meant "provide a mechanism to run these tests in DejaGNU, 
optionally."  Probably by means of a site.exp variable.

> In general, I'd like to see the focus on making installation testable.
> Let's start there! Wholesale conversion of specialized libstdc++ make
> check-* rules to conform to dejagnu's scary acid-trip can be undertaken
> later, huh? 

I agree.

Let me be very concrete.

The very next steps I would like to take are:

0. Resolve whatever normal.exp problem you're having, so that I can make 
my promised update the online docs to reflect the current situation.  I 
can do the doc updates anyhow, but it seems a bit pointless until we 
have some independent confirmation that my machine is not magically blessed.

1. Modify DejaGNU to build the .po files at test-time, without making 
any changes to the Makefiles, etc.  This will fix Paolo's bug, and will 
make a handful of tests that presently fail when testing an installed 
compiler (due to lack of the .po files) work.

At that point, I think we'll have achieved the bare minimum goal: making 
it possible to run the testsuite on an installed compiler, and get the 
same results you get in the build directory.  So, we could stop there.

However, it would be nice to do some additional things that would 
eliminate redundancy and make it possible to run the other testsuite 
variations (like compile-only and performance) on installed compilers.

Potential next steps would be:

1. Move check_performance/check_compile functionality into DejaGNU, as 
optional modes.  Modify Makefile targets to use DejaGNU to do this, 
rather than run the existing scripts.  Remove the existing scripts. 
(This could be done as three independent pieces, one per sentence.)

2. At this point, I think that there will no longer be any references to 
libv3test.a, outside of DejaGNU.  Thus, the Makefile rules to build it 
could be removed.  This is just a cleanup.

3. The create_testsuite_files is now redundant with logic in DejaGNU. 
Therefore, we could remove the script.  Or, alternatively, we could have 
DejaGNU create the files presently created by create_testsuite_files. 
(This is independent of whether we *use* testsuite_files if it exists; 
I'm merely talking about how it gets created.)

-- 
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com
(916) 791-8304