Automated testing framework (was: V3: SPARC bug and tree freeze)

[Laurent Guerby <guerby at acm dot org>]

Robert Dewar wrote:
> Potentially yes, it is a bunch of scripts that would need quite a bit
> of massaging for general use. This might be something that Laurent
> Guerby could help with.

I would be happy to do so. 

For the curious, the ACT scripts do roughly as follows (well at least
2 years ago when I worked on them ;-):

1. Checkout and package the nigthly source tarball (at some fixed
hour) on one machine.

2. On various architectures, completely build and user install GNAT
from the source tarball and using a stable compiler).

3. Run ACATS and internal test suite to provide test result baseline
and send a detailed report of nightly changes and failures.

4. On various architectures a server wait for patches, apply the
patches, rebuild and install the compiler, build and run the internal
test suite, produces a report of changes against the nightly baseline,
and goes back to clean state. The server interface is email based, and
requests are queued.

Note that since the system starts out from fresh sources, it usually
is able to catch build stuff regression (missing files, wrong
Makefile dependancies, etc...).

I didn't look too closely at the current GCC testing framework (seems
like dejagnus is a huge beast, but embedded build and run annotations
for it look simple to grok anyway - good design ;-), and I don't know
what C/C++/other language packages are reasonable candidates for
filling the test suite (ie they have lots of testing executables).

All this stuff is not rocket science, but is definitely a big plus to
have and even more so if a project reaches a point when 100% pass is
required before commit.

News on the Ada testing front:

- Thanks to ACT contribution of configuration files, I'm now able to
build and run 2361 executable ACATS tests with the latest public GNAT
release (GCC 2.8.1 based), 16 executable tests are missing, mostly
distributed tests that have exotic run requirements (total build and
run time is 45 minutes on a P2 350MHz).

- The ACES performance testing suite packaging effort is frozen
pending copyright investigation (the government agency that contracted
the thing no longer exists).

- ACT is working on selecting and fixing the copyright/license on a
subset of their internal test, and will provide them to me for
packaging.

This means that the GNAT testing framework should be there in time
when the GNAT sources commit happens. I hope it can be used by people
to build and test GNAT on platforms ACT doesn't support yet like
GNU/Linux on Alpha, SPARC and PowerPC.

BTW, could someone send me the legal form(s) for GCC contribution
copyright assignement? I'm in working in France for a bank, have no
obnoxious clause in my contract, and work on the GNU project on my
free time with my home machine and email (both completely separated
from my day work envrionment).

-- 
Laurent Guerby <guerby@acm.org>