Re: [v3] make check-abi

[Janis Johnson <janis187 at us dot ibm dot com>]

On Fri, Aug 23, 2002 at 09:35:58AM -0700, Benjamin Kosnik wrote:
> 
> > Will you have a provision for baseline data that depends on how the
> > library is configured, e.g., thread support?
> 
> Not at the moment, although that might be something to look at in the future.

See comments below about why we should think about this now.

> > The tests in gcc/testsuite/consistency.vlad also need baseline files.
> > Have any of you looked at those to see if the same mechanism can be
> > used?  Once again, there are compiler options that affect the results,
> > so there should be a provision for multiple baselines per target.
> > 
> > Has anyone tried running the consistency.vlad tests recently?  I can run
> > them with hacks to various .exp files, but I don't yet understand the
> > tools well enough to know what I'm doing.  If someone is already fixing
> > these, great, otherwise I'll keep plugging along so we can start using
> > those tests.
> 
> I have no idea what to do with those tests. I'm primarily concerned with
> the library aspects of the C++ ABI. The compiler bits also need a full
> set of testing. I'm not quite sure who, if anyone, is working on that.

I'm working on that and am hoping that others will as well.

I'm close to having the consistency.vlad tests working.  There were some
changes to the testsuite a few months ago that broke them, and it turned
out to be fairly straightforward to fix the vlad.exp file.

The hard part is figuring out what to do with the baseline result files.
Many of the same issues apply to your library ABI tests, so it would
help to figure them out soon so we can use the same mechanism.

- How should the baseline results be identified?  The default could be
  $target_triplet, but sometimes it will need to be broken down further,
  as when someone wants to test consistency for special options that
  affect the test output.  There are ways to configure the library that
  change the results of your library tests, and there are also ways to
  compile that affect layout and alignment.  If these are common and
  should be checked, then people who want those baselines can determine
  names for them and define how they are built.  Similarly, there might
  be sets of baseline results that are expected to be identical even
  though $target_triplet is different.  For example, the 3.2 build
  status list includes powerpc-ibm-aix4.2.1.0, powerpc-ibm-aix4.3.2.0,
  powerpc-ibm-aix4.3.3.0, and powerpc-ibm-aix5.1.0.0.  I suspect that
  some or all of those could use the same baseline results for both the
  library tests and the compiler tests.

- How should the baseline results be stored?  For the layout tests in
  consistency.vlad, keeping a tarball would save a lot of space, which
  will be important when there are lots of sets of baseline results:

    laptop% !du
    du -s i686*
    1097    i686-pc-linux-gnu-results
    36      i686-pc-linux-gnu-results.tar.gz

  Could the test suite handle unpacking a needed tarball and leaving
  the others alone?

- How should the desired baseline results be specified?  Currently the
  consistency.vlad tests look for BASE_COMPILER (not a great name),
  which is part of the name of the directory containing the results.
  The default could be determined with configury magic, and could be
  used for the library tests as well as the compiler tests.

- Will we want compiler consistency tests for languages other than C++?
  If so, we'll need a better way to organize the consistency tests.

- Will we want a single make check target that covers both language and
  compiler consistency/ABI testing?

Janis