This is the mail archive of the
gcc-patches@gcc.gnu.org
mailing list for the GCC project.
Re: RFC: run tests on installed compiler
Benjamin Kosnik wrote:
The V3 testsuite was (long ago) converted to DejaGNU from its previous
format. Perhaps for this reason, the testsuite was not set up to work
well with installed compilers.
What did you have in mind? If you can be specific about what kind of
process you are aiming for it would be helpful. So you have an installed
toolchain. What else to you need to test it? What kind of process are
you aiming for?
My high-level goal is to be able to validate the actual compilers that
we provide to customers, in the same environment that the user will use
them.
Technically, what I want to be able to do is the following three-step
process:
1. Put compilers I want to test in my PATH. Set LD_LIBRARY_PATH, if I
need to do that to run programs copmiled with this compiler. In other
words, set things up as I would if I were just going to use this
compiler to build an application.
2. Create a site.exp file that contains only the host, build, and target
machine triplets.
3. Run:
runtest --tool libstdc++ --srcdir=/path/to/libstdc++-v3/testsuite
I want that command to run the entire V3 testsuite to be run, and
produce the exact same results that I would get from "make check" in the
build tree.
This exact procedure already works for (at least) Binutils, GAS, GCC,
and G++. (It doesn't quite work for GNU ld at the moment, but I have a
patch for that which I'll be submitting to binutils.)
There is a rule (make check-script, make check-script-install) and a
crufty script that allow checking of installed libraries, both static
and shared.
Is this the kind of thing you want to do, but better? Sounds cool.
Yes, that's the idea. However, I want to be able to use DejaGNU both
for consistency with the rest of the toolchain, and for its ability to
handle cross environments. I want to be able to do this with no trace
of the build environment around, because I want to prove to myself that
there's absolutely no accidental pollution coming from that.
Is the g++ testsuite testable like this? If so, let's do something similar.
Yes, the G++ testsuite works exactly like this. The fundamental
requirement is that the testsuite has to avoid depending on anything
built during the build process that is not installed.
Of course, as Mike suggested, it can make use of those things
optionally, but, as Dan and I pointed out, you really want to avoid
that, because it's likely to eventually result in differences in
testsuite runs between testing in the build-tree and testing with
installed toolchains.
(Even automatically reading testsuite_files if it exists is potentially
a problem in this way, in that it means that the non-testsuite_files
codepath will probably be rarely used, and therefore more likely to be
in disrepair. However, in my opinion, reading testsuite_files if it
exists is much less risky than, for example, building libv3test.a at
build time, which will result in duplicative logic to actually build the
library.)
I would like to see a toplevel make rule, called "make check-install"
that does this for the entire toolchain. It's ok if we don't get there
all at once, but I think we should keep that in mind.
I think it belongs in the existing "contrib/test_installed" instead.
You can copy that script to your test machine and run it; it's not
desirable to have any part of the build machinery around. I suppose we
could have "make check-install" run "test_installed", if that was useful.
In this patch, I fixed the most important cases of this problem: the
use of the testsuite_files file, and the use of libv3test.a. (In the
case of the former, we now generate the list of tests dynamically; in
the case of the latter, we build the objects at test-time.) I did not
correct the handling of fr.po and de.po; these, too, should be built
at test time, with the result that LOCALEDIR should just be the
current directory. I will work on that in the near future.
My preference is to allow use of testsuite_file, but make it optional,
so that in its absence, the list of files is autogenerated. I believe
this is the preference of Phil, and Mike Stump as well. Perhaps others
have a preference too: if so, this is probably a good time to say so.
I would prefer that testsuite_file not be used, but I will defer to your
instructions. Right now, I'm interpreting what you've said as an
opinion, not a firm direction to change the way it works. If you want
me to change it, please tell me do that explicitly.
The rest I have no problem with. (ie, LOCALEDIR, building v3test.a at test time.)
Excellent.
1. Remove check_compile, in favor of providing an option to DejaGNU to
just generate assembly files, if people still need this
functionality.
Suggestion: come up with the dejagnu magic to do this, and then talk about
removing this rule. If you think this functionality is of more general
use than just libstdc++, and want to push it everywhere: cool. Keep the
rule though, it's useful.
OK. I think this is easy to do.
2. Modify check_performance to work without testsuite_files, probably
by integrating that functionality with DejaGNU.
Have you ever run this rule? BTW it uses testsuite_files_performance.
Yes, I understand, and by "integrate" I did not mean "run these tests by
default". I meant "provide a mechanism to run these tests in DejaGNU,
optionally." Probably by means of a site.exp variable.
In general, I'd like to see the focus on making installation testable.
Let's start there! Wholesale conversion of specialized libstdc++ make
check-* rules to conform to dejagnu's scary acid-trip can be undertaken
later, huh?
I agree.
Let me be very concrete.
The very next steps I would like to take are:
0. Resolve whatever normal.exp problem you're having, so that I can make
my promised update the online docs to reflect the current situation. I
can do the doc updates anyhow, but it seems a bit pointless until we
have some independent confirmation that my machine is not magically blessed.
1. Modify DejaGNU to build the .po files at test-time, without making
any changes to the Makefiles, etc. This will fix Paolo's bug, and will
make a handful of tests that presently fail when testing an installed
compiler (due to lack of the .po files) work.
At that point, I think we'll have achieved the bare minimum goal: making
it possible to run the testsuite on an installed compiler, and get the
same results you get in the build directory. So, we could stop there.
However, it would be nice to do some additional things that would
eliminate redundancy and make it possible to run the other testsuite
variations (like compile-only and performance) on installed compilers.
Potential next steps would be:
1. Move check_performance/check_compile functionality into DejaGNU, as
optional modes. Modify Makefile targets to use DejaGNU to do this,
rather than run the existing scripts. Remove the existing scripts.
(This could be done as three independent pieces, one per sentence.)
2. At this point, I think that there will no longer be any references to
libv3test.a, outside of DejaGNU. Thus, the Makefile rules to build it
could be removed. This is just a cleanup.
3. The create_testsuite_files is now redundant with logic in DejaGNU.
Therefore, we could remove the script. Or, alternatively, we could have
DejaGNU create the files presently created by create_testsuite_files.
(This is independent of whether we *use* testsuite_files if it exists;
I'm merely talking about how it gets created.)
--
Mark Mitchell
CodeSourcery, LLC
mark@codesourcery.com
(916) 791-8304