Re: C++ compatibility testing

[Janis Johnson <janis187 at us dot ibm dot com>]

On Sat, Sep 21, 2002 at 02:36:54PM -0400, Phil Edwards wrote:
> On Fri, Sep 20, 2002 at 02:58:54PM -0700, Janis Johnson wrote:
> > Phil and Benjamin,
> > 
> > I'm again looking into how to set up tests within the GCC testsuite to
> > build two pieces of a C++ test using different versions of GCC and then
> > linking them and running the test.  Phil, you mentioned that you might
> > try this (http://gcc.gnu.org/ml/gcc/2002-08/msg01259.html); please let
> > me know if you're doing something similar.
> 
> Er, well, best laid plans...
> 
>     http://gcc.gnu.org/ml/libstdc++/2002-09/msg00119.html

Ouch.  I'll stick to juggling.

> 
> > If you've got a scheme for
> > specifying a second compiler for the test tools to use, let me know what
> > it is.
> 
> When you posted your script to do this, you wrote:
> 
> > They required either building
> > object files with older compilers ahead of time and then using those
> > archived files when testing the new compiler, or else having both old
> > and new compilers available when running the tests.
> 
> I will be using the second approach, since I already have both compilers:
> 
>     http://gcc.gnu.org/ml/gcc/2002-08/msg01133.html
> 
> For each night, I can just run the autocrasher, then I have both branches
> available in an easily computable pathname:
> 
>     27% ls -d build/*2002-09-21*
>     build/build-2002-09-21/        build/install-2002-09-21/
>     build/build-2002-09-21-3.2/    build/install-2002-09-21-3.2/
>     build/build-2002-09-21-trunk/  build/install-2002-09-21-trunk/
>     28%
> 
> So, using a variant of your script, OLDCXX is install-(the_date)-3.2/bin/g++
> and NEWCXX is install-(the_date)-trunk/bin/g++.
> 
> I still have not thought of a decent way to do this inside the gcc testsuite.
> It may be that we need to use the first approach for general testing,
> and have some .o's sitting around for people.

I'd like to somehow specify the installation location of the second
compiler to use, probably as an environment variable.  Keeping object
files around is just too messy.
 
> > After I've got things working with building simple relocatable objects
> > and linking them together, I'll try building the pieces of a test into
> > static and/or shared libraries in the hopes of finding different kinds
> > of problems.  That might be messy using dejagnu, though, and I only have
> > GNU/Linux systems on which to try out my changes.
> 
> You might (or might not) find it easier to use the QMTest thingy that
> the Software Carpentry slash CodeSorcery folks are putting together.
> I made one pass, but apparently my version of Python was too recent.
> At any rate, making "unusual" configurations work is supposed to be much
> easier in QMTest than in DejaGNU.

I tried QMTest last May and had problems, probably because I was using
Python 2.2 which doesn't seem to work with it.  Now I've got it working
on i686-pc-linux-gnu and ia64-unknown-linux-gnu with:

  Python 2.1.3
  QMTest 1.1.5
  QMTC 1.2

It might have good support for this kind of testing, but I'm not clear
about where to add the new functionality needed in the test classes.
Those test classes are in QMTC, which is copyrighted by CodeSourcery and
resides at their ftp site, not in the GCC sources.  I would need
permission from my employer to contribute to that package, and probably
some kind of copyright agreement with CodeSourcery.

Using QMTest for the interoperability tests won't be practical unless
GCC developers who want to run those tests are set up to use QMTest.
Are they, and if not, will they be willing to set it up so they can run
the new tests?

Janis