This is the mail archive of the gcc-patches@gcc.gnu.org mailing list for the GCC project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: RFC: run tests on installed compiler


Benjamin Kosnik wrote:

The V3 testsuite was (long ago) converted to DejaGNU from its previous
format.  Perhaps for this reason, the testsuite was not set up to work
well with installed compilers.


What did you have in mind? If you can be specific about what kind of
process you are aiming for it would be helpful. So you have an installed
toolchain. What else to you need to test it? What kind of process are
you aiming for?

My high-level goal is to be able to validate the actual compilers that we provide to customers, in the same environment that the user will use them.


Technically, what I want to be able to do is the following three-step process:

1. Put compilers I want to test in my PATH. Set LD_LIBRARY_PATH, if I need to do that to run programs copmiled with this compiler. In other words, set things up as I would if I were just going to use this compiler to build an application.

2. Create a site.exp file that contains only the host, build, and target machine triplets.

3. Run:

runtest --tool libstdc++ --srcdir=/path/to/libstdc++-v3/testsuite

I want that command to run the entire V3 testsuite to be run, and produce the exact same results that I would get from "make check" in the build tree.

This exact procedure already works for (at least) Binutils, GAS, GCC, and G++. (It doesn't quite work for GNU ld at the moment, but I have a patch for that which I'll be submitting to binutils.)

There is a rule (make check-script, make check-script-install) and a
crufty script that allow checking of installed libraries, both static
and shared.

Is this the kind of thing you want to do, but better? Sounds cool.

Yes, that's the idea. However, I want to be able to use DejaGNU both for consistency with the rest of the toolchain, and for its ability to handle cross environments. I want to be able to do this with no trace of the build environment around, because I want to prove to myself that there's absolutely no accidental pollution coming from that.


Is the g++ testsuite testable like this? If so, let's do something similar.

Yes, the G++ testsuite works exactly like this. The fundamental requirement is that the testsuite has to avoid depending on anything built during the build process that is not installed.


Of course, as Mike suggested, it can make use of those things optionally, but, as Dan and I pointed out, you really want to avoid that, because it's likely to eventually result in differences in testsuite runs between testing in the build-tree and testing with installed toolchains.

(Even automatically reading testsuite_files if it exists is potentially a problem in this way, in that it means that the non-testsuite_files codepath will probably be rarely used, and therefore more likely to be in disrepair. However, in my opinion, reading testsuite_files if it exists is much less risky than, for example, building libv3test.a at build time, which will result in duplicative logic to actually build the library.)

I would like to see a toplevel make rule, called "make check-install"
that does this for the entire toolchain. It's ok if we don't get there
all at once, but I think we should keep that in mind.

I think it belongs in the existing "contrib/test_installed" instead. You can copy that script to your test machine and run it; it's not desirable to have any part of the build machinery around. I suppose we could have "make check-install" run "test_installed", if that was useful.


In this patch, I fixed the most important cases of this problem: the
use of the testsuite_files file, and the use of libv3test.a.  (In the
case of the former, we now generate the list of tests dynamically; in
the case of the latter, we build the objects at test-time.)  I did not
correct the handling of fr.po and de.po; these, too, should be built
at test time, with the result that LOCALEDIR should just be the
current directory.  I will work on that in the near future.

My preference is to allow use of testsuite_file, but make it optional, so that in its absence, the list of files is autogenerated. I believe this is the preference of Phil, and Mike Stump as well. Perhaps others have a preference too: if so, this is probably a good time to say so.

I would prefer that testsuite_file not be used, but I will defer to your instructions. Right now, I'm interpreting what you've said as an opinion, not a firm direction to change the way it works. If you want me to change it, please tell me do that explicitly.


The rest I have no problem with. (ie, LOCALEDIR, building v3test.a at test time.)

Excellent.


1. Remove check_compile, in favor of providing an option to DejaGNU to
  just generate assembly files, if people still need this
  functionality.


Suggestion: come up with the dejagnu magic to do this, and then talk about
removing this rule. If you think this functionality is of more general
use than just libstdc++, and want to push it everywhere: cool. Keep the
rule though, it's useful.

OK. I think this is easy to do.


2. Modify check_performance to work without testsuite_files, probably
  by integrating that functionality with DejaGNU.


Have you ever run this rule? BTW it uses testsuite_files_performance.

Yes, I understand, and by "integrate" I did not mean "run these tests by default". I meant "provide a mechanism to run these tests in DejaGNU, optionally." Probably by means of a site.exp variable.


In general, I'd like to see the focus on making installation testable.
Let's start there! Wholesale conversion of specialized libstdc++ make
check-* rules to conform to dejagnu's scary acid-trip can be undertaken
later, huh?

I agree.


Let me be very concrete.

The very next steps I would like to take are:

0. Resolve whatever normal.exp problem you're having, so that I can make my promised update the online docs to reflect the current situation. I can do the doc updates anyhow, but it seems a bit pointless until we have some independent confirmation that my machine is not magically blessed.

1. Modify DejaGNU to build the .po files at test-time, without making any changes to the Makefiles, etc. This will fix Paolo's bug, and will make a handful of tests that presently fail when testing an installed compiler (due to lack of the .po files) work.

At that point, I think we'll have achieved the bare minimum goal: making it possible to run the testsuite on an installed compiler, and get the same results you get in the build directory. So, we could stop there.

However, it would be nice to do some additional things that would eliminate redundancy and make it possible to run the other testsuite variations (like compile-only and performance) on installed compilers.

Potential next steps would be:

1. Move check_performance/check_compile functionality into DejaGNU, as optional modes. Modify Makefile targets to use DejaGNU to do this, rather than run the existing scripts. Remove the existing scripts. (This could be done as three independent pieces, one per sentence.)

2. At this point, I think that there will no longer be any references to libv3test.a, outside of DejaGNU. Thus, the Makefile rules to build it could be removed. This is just a cleanup.

3. The create_testsuite_files is now redundant with logic in DejaGNU. Therefore, we could remove the script. Or, alternatively, we could have DejaGNU create the files presently created by create_testsuite_files. (This is independent of whether we *use* testsuite_files if it exists; I'm merely talking about how it gets created.)

--
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com
(916) 791-8304


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]