Re: RFC: run tests on installed compiler

[Benjamin Kosnik <bkoz at redhat dot com>]

> My high-level goal is to be able to validate the actual compilers that 
> we provide to customers, in the same environment that the user will use 
> them.
> 
> Technically, what I want to be able to do is the following three-step 
> process:
> 
> 1. Put compilers I want to test in my PATH.  Set LD_LIBRARY_PATH, if I 
> need to do that to run programs copmiled with this compiler.  In other 
> words, set things up as I would if I were just going to use this 
> compiler to build an application.
> 
> 2. Create a site.exp file that contains only the host, build, and target 
> machine triplets.
> 
> 3. Run:
> 
> runtest --tool libstdc++ --srcdir=/path/to/libstdc++-v3/testsuite
> 
> I want that command to run the entire V3 testsuite to be run, and 
> produce the exact same results that I would get from "make check" in the 
> build tree.

This sounds good to me. I suppose I could check static linkage testresults with the appropriate RUNTEST flags too.

Then the check-script and check-script install bits could be removed,
and this (more sane) test procedure could be documented.

> > I would like to see a toplevel make rule, called "make check-install"
> > that does this for the entire toolchain. It's ok if we don't get there
> > all at once, but I think we should keep that in mind.
> 
> I think it belongs in the existing "contrib/test_installed" instead. 
> You can copy that script to your test machine and run it; it's not 
> desirable to have any part of the build machinery around.  I suppose we 
> could have "make check-install" run "test_installed", if that was useful.

I like the idea of "make check-install" running test_installed. This
part doesn't matter so much to me though.

> > Suggestion: come up with the dejagnu magic to do this, and then talk about
> > removing this rule. If you think this functionality is of more general
> > use than just libstdc++, and want to push it everywhere: cool. Keep the
> > rule though, it's useful.
> 
> OK.  I think this is easy to do.

Sounds good.

> >>2. Modify check_performance to work without testsuite_files, probably
> >>   by integrating that functionality with DejaGNU.
> > 
> > 
> > Have you ever run this rule? BTW it uses testsuite_files_performance.
> 
> Yes, I understand, and by "integrate" I did not mean "run these tests by 
> default".  I meant "provide a mechanism to run these tests in DejaGNU, 
> optionally."  Probably by means of a site.exp variable.

Sounds good.

Again, don't care about the exact mechanism as long as there is a "make
check-performance" that works.

> Let me be very concrete.
> 
> The very next steps I would like to take are:
> 
> 0. Resolve whatever normal.exp problem you're having, so that I can make 
> my promised update the online docs to reflect the current situation.  I 
> can do the doc updates anyhow, but it seems a bit pointless until we 
> have some independent confirmation that my machine is not magically blessed.

Taken care of, thanks for your hard work.

I'm not quite sure what's up with my system (stock FC3). I can't get the gcc tests to work either.
 
> 1. Modify DejaGNU to build the .po files at test-time, without making 
> any changes to the Makefiles, etc.  This will fix Paolo's bug, and will 
> make a handful of tests that presently fail when testing an installed 
> compiler (due to lack of the .po files) work.

Sounds great.
 
> Potential next steps would be:
> 
> 1. Move check_performance/check_compile functionality into DejaGNU, as 
> optional modes.  Modify Makefile targets to use DejaGNU to do this, 
> rather than run the existing scripts.  Remove the existing scripts. 
> (This could be done as three independent pieces, one per sentence.)

Ok provided the makefile rules stay and there is still testsuite_files_performance.

> 2. At this point, I think that there will no longer be any references to 
> libv3test.a, outside of DejaGNU.  Thus, the Makefile rules to build it 
> could be removed.  This is just a cleanup.

Sounds good.
 
> 3. The create_testsuite_files is now redundant with logic in DejaGNU. 
> Therefore, we could remove the script.  Or, alternatively, we could have 
> DejaGNU create the files presently created by create_testsuite_files. 
> (This is independent of whether we *use* testsuite_files if it exists; 
> I'm merely talking about how it gets created.)

Eh, less keen.

-benjamin