ACATS

[<guerby at acm dot org>]

[I have ISP problems at the moment, I hope this will get through.]

[0] General:

Jerry: Basically the question is if you want to see the test suite as
an scaled down ACATS (in which case all changes etc need to be
documented), or see ACATS as a starting point, and develop the suite
further, for example adding regression testing for discovered bugs,
etc. My first impulse is to stick to ACATS, to follow language
development, and add extra tests separately.

self: ACATS in GCC will be the part of ACATS where a volunteer has
done the packaging work. I proposed to do a piece of this packaging,
if people want to work on packaging more I'll try to help but won't do
it myself, I certainly agree that packaging all of ACATS is a great
goal, as I said I'll carefully document what I left out and why,
anyone is free to finish the work. IMHO non ACATS or ACATS-derived
tests should be packaged in another subdirectory so that it is easier
to track upstream ACATS.  The December ACATS update email has been
sent, I'm forwarding it to the list so people can make an
idea. Following upstream ACATS should be very easy, all changes are
very well documented and there isn't a large volume of changes.

Joseph S. Myers: We ought to have a link to this from the Ada section
of readings.html.

self: I'll work hard at documenting and linking to a maximum of Ada
ressources, Ada is pretty unique in that all (or nearly all) language
requirement, design, norms, maintainance and testsuite sources and
documents are freely available.

[1] pre-gnatchop

Jerry: I assume we want to run an ACATS subset as a basic sanity
check, right ?  But even in that case, it would be advisable to follow
the changes made to the test suite. Your proposal means this has to be
done manually by someone. Who is going to monitor this, and propagate
all changes in the tests ?

self: I proposed to do the upstream tracking work for the executable
section. As for the subset we want to run, this is open to
discussions, I have no opinion on the subject. Personally, I'll run
every single GCC test available for all of my changes up to 8 hours of
wall clock time (make a daily patch, check during night, commit in the
morning).

[2] directory structure

Mike Stump: Split them up some.  <200 per directory would be good (or
maybe <359 per directory).

Jerry: I like the current subdirectory structure, it makes it easier
if you want to check just one test, or a set or related tests.

Geert: From experience I find it really valuable to have separate
subdirectories per class/chapter. For example, if you work on some
changes to handling of floating point, you would initially just want
to run chapter CXG, which means the executable RM Annex G tests.

self: okay, I'll keep the existing ACATS directory structure.  Running
easily subsets of scripts is on the spec of any testing technology and
shouldn't be related to directory structure. I intend to have the test
list file provide test key annotated with a few attributes, like
wether the test use tasking, is ok for the no-run-time mode, has some
real code, is part of minimal testing requirement, etc... so that it
is easy select a test subset.

[3] filename length

Geert: It might be preferable to "krunch" the names to a more
managable length using the -gnatk option, since there are still many
filesystems that have issues with long names and it's also easier to
deal with shorter names as human beings (those who have seen the few
examples of few long names in ACATS tests, will agree they do not
help: only the first few are useful).

self: I believe the longest name is
ca13001_1-ca13001_2-ca13001_4-family_transportation.adb 

There is a test checking for the longest package name supported by the
implementation, this is part of my "drop because stupid and painful"
list. Such tests should be auto-generated and tailored to GCC in some
other part of the test suite, the ACATS stuff is of little value.

Mike Stump: I'd say ok.

self: let's say okay.

[4] top dir gcc/gcc/testsuite/ada/acats

Joseph S. Myers: That seems plausible - presumably "runtest --tool
ada" will be used in "make check" to run both those and other Ada
tests?

self: to be decided.

[5] source separate from harness

self: no comment, okay.

[6] drop compilation model tests

self: Geert and Robert agreed, okay.

[7] B tests

Mike Stump: They can be made to require less, by having them line
resolution only, this is what the C++ testsuite does.  This is for the
Ada maintainers to determine.  If they want the testing, and want the
burden, then they should stay the same, if they want to escape it,
then a line based approach can be used.

Jerry: I think it's important to test the whole GNAT system, not just
the backend.  The value of the B tests is that it alerts you to
unexpected changes. I think that at least for now they should stay. We
can add 'skip tests' for changes that are to much work for the added
value.

Zack Weinberg: I disagree in the strongest possible terms.  Put the B
tests in the public repository.  If you don't, you only make life
harder for people outside of ACT who wish to work on the Ada front
end.  The maintenance work has to be done anyway, and ought to be the
responsibility of the person who makes the change that causes the
tests to regress.  If the B tests are run as part of "make check" in
the FSF tree, this will be enforced automatically.

self: B tests are 1525 files, 11886 errors to be flagged (errors are
marked with a -- ERROR: xxx comment).  No one is against inclusion, I
just points out that there is painful initial work and maintainance,
and that I don't volunteer to do it, given passionate answers, I
assume that Zack and Jerry volunteered to do the work :). I will also
point out that actually including this might make it less likely that
people will propose patches to improve error messages, since any of
such patches will probably generate hundreds of changes to be
validated in the B tests, this is not really fun, so by this effect
including B test requirement might *lower* the future quality of the compiler
by preventing some useful patches.  Also note that GNAT currently
generates the best error messages I've ever seen for any compiler and
language, and by a fair margin.

[8] tasking delay patch

Geert and Richard Kenner: Test timings?

self: ACATS timing on a 1GHz P3 at -O0, in number of test and seconds
per section:

   N ; T(s) ; Desc.

   4 ; 	30  ; support + cz Check that the test reporting stuff works
  75 ; 	38  ; a  
  34 ; 	15  ; c2 Lexical Elements
 351 ; 267  ; c3 Declarations and Types
 339 ; 186  ; c4 Names and Expressions
  95 ; 	59  ; c5 Statements
  81 ; 	43  ; c6 Subprograms 
  51 ; 	48  ; c7 Packages
 140 ; 141  ; c8 Visibility Rules
 255 ; 307  ; c9 Tasks and Synchronization
  74 ; 	49  ; ca Program Structure and Compilation Issues
  43 ; 	26  ; cb Exceptions
 117 ; 	78  ; cc Generic Units
 173 ; 	87  ; cd Representation Issues
 268 ; 298  ; ce Predefined Language Environment (Ada 83)
  87 ; 133  ; cxa Predefined Language Environment (Ada 95)
  30 ; 	32  ; cxb Interface to Other Languages
  13 ; 	30  ; cxc Systems Programming
  38 ; 	75  ; cxd Real-Time Systems
   1 ; 	 1  ; cxe Distributed Systems 
  20 ; 	24  ; cxf Information Systems
  29 ; 	63  ; cxg Numerics
   4 ; 	19  ; cxh Safety and Security
   4 ; 	 2  ; d  
  11 ; 	 5  ; e  
  26 ; 	19  ; l  

Joseph S. Myers: It is arguable that perhaps the CVS vendor branch
mechanism should be used - import the unmodified sources on the vendor
branch with GCC's modified version on the mainline.  This should make
it easy to use CVS to see what the differences from unmodified ACATS
are.  I don't know whether whatever tests it is deliberately decided
to drop rather than include in the GCC testsuite should also go on the
vendor branch, or whether it should just be documented which are
dropped.

self: I'm no CVS specialist, keeping things in ACATS original form
will require maintaining piles of dubious scripts at best, I don't
want to do that myself, I by far prefer the hand crafted approach, and
have things directly in GCC usable form in CVS.

[9] prototype tarball ftp upload

self: I'll put stuff in <http://perso.wanadoo.fr/guerby/acats/>

[10] first harness prototype

Geert: You did not really address the reason why you had decided not
to use the existing testing harness. I can see a number of reasons why
seperate scripts would be better for ACATS, but these should be
documented.

Geoff Keating: You do plan to use dejagnu to run the tests, correct?

self: My initial tarball will only contain readily compilable sources,
a list of test main names and attributes as described above,
documentation on the test to run and omitted, and a minimalistic
Bourne shell script to run the thing.  From there, I assume DejaGNU
experts will volunteer or guide volunteers on how to best integrate
this in the existing Makefile + dg scripts. I must admit I'm not
really impressed by the existing testing framework, and I believe and
I can do much better with not that much effort, I'll propose something
later on, but I don't want this to delay inclusion of ACATS in usable
form in the GCC project, and I certainly don't want to stop people
that would volunteer on integrating ACATS and existing DejaGNU.
People will also have to decide what is officially part of the
mandatory testing process.

[12] DFAR legalese and GCC changes

Geert: I wouldn't think so. This seems to only add confusion about the
licensing.  Of course if we would make significant changes to the
tests, we should put the files under GPL. Also any new files and
scripts should be GPL-ed.

self: when I'll gnatchop the original sources, only one file will
retain the legalese, I assume I don't need to copy it to every file
after gnatchop.

Mike Stump: No.

Hmm, I don't intend to change anything without properly identifying
what I've done (like any other change made to any part of GCC), may be
I misunderstand what your answer means?

[13] maintainership

self: I'll add myself in MAINTAINERS for gcc/gcc/testsuite/ada/acats
once the discussion is complete and the commit is done.

-- 
Laurent Guerby <guerby@acm.org>