The Ada language lacks a test suite. Some level of testing occurs during bootstrap since the front end and libraries are written in Ada, but this is inadequate. There has been some discussion of wrapping portions of the ACATS validation suite into the DejaGNU framework. Framework for a regression suite along the lines of c-torture should also be established. Release: 3.1 20020308 (prerelease)
Responsible-Changed-From-To: unassigned->guerby Responsible-Changed-Why: Laurent volunteered setting up the ACATS test suite.
Any news about the Ada testsuite? Is ACATS likely to make the gcc testsuite before 3.4? Thanks.
ACATS won't make it into 3.4, you can still run it by downloading it from <http://perso.wanadoo.fr/guerby/> and following the instructions, current version as of 20030706 is 0.7.
Ada really should have a test suite so that changes to the backend get exersied on Ada.
I've started from Laurent's set up to integrate the ACATS test suite as part of 'make check'. I have a complete set up which is working. Since Laurent and I know nothing about dejagnu, we're simply using sh to run the test suite, which is working just fine. The integration follows other gcc requirements: using make check will run the acats test suite if Ada is enabled, and make check-ada will work on a non installed version of the compiler. If this is acceptable, the ACATS test suite along with the makefile changes could be incorporated as soon as practical (e.g. probably after the 3.4 branch). I am using make check-ada (part of make check) before any of my check ins. Arno
Subject: Re: Ada has no test suite. On Wed, 22 Oct 2003, charlet at gcc dot gnu dot org wrote: > I have a complete set up which is working. Since Laurent and I know nothing > about dejagnu, we're simply using sh to run the test suite, which is > working just fine. But is it working for testing cross-compilers where programs for the target need to be run on a simulator, or on an embedded board, or ...? And does it generate .sum files in the Dejagnu format for the use of the automated regression testers (this point is critical to get much of the benefit of the testing, a sh script can always be committed first then converted to a Dejagnu harness later to take advantage of all the support Dejagnu has for testing cross-compilers)? Does your test setup support additionally "ordinary" testsuite directories for gradually accumulating tests in the style of gcc.c-torture/{compile,execute} (test compilation / execution works with multiple optimization options) or gcc.dg (test that error/warning messages are as expected)? There are many such tests that Kenner has posted to the lists over the years (for back-end bugs shown up by Ada testcases) that should be gathered into the testsuite.
<< But is it working for testing cross-compilers where programs for the target need to be run on a simulator, or on an embedded board, or ...? >> Yes, it has support for cross testing, although I haven't used it myself. Maybe Laurent can comment on this part. Basically there's a customization section where you can specify how to load/run an executable. << And does it generate .sum files in the Dejagnu format for the use of the automated regression testers (this point is critical to get much of the benefit of the testing, a sh script can always be committed first then converted to a Dejagnu harness later to take advantage of all the support Dejagnu has for testing cross-compilers)? >> No, but if there's a clear documentation about the format of .sum files, I'd guess this should not be very hard to do. << Does your test setup support additionally "ordinary" testsuite directories for gradually accumulating tests in the style of gcc.c-torture/{compile,execute} (test compilation / execution works with multiple optimization options) or gcc.dg (test that error/warning messages are as expected)? >> The current set up only supports executable tests, but it shouldn't be too hard to enhance it if needed. Arno
Subject: Re: Ada has no test suite. > On Wed, 22 Oct 2003, charlet at gcc dot gnu dot org wrote: > > > I have a complete set up which is working. Since Laurent and I know nothing > > about dejagnu, we're simply using sh to run the test suite, which is > > working just fine. If someone is willing to contribute a dejagnu Ada harness, we're ready to use it, otherwise we'll do with simple scripts. > But is it working for testing cross-compilers where programs for the > target need to be run on a simulator, or on an embedded board, or ...? Joel Sherrill did run my original script with a cross-compiler on a simulator and IIRC he had trouble running dejagnu (much more complicated to set up). > > And does it generate .sum files in the Dejagnu format for the use of the > automated regression testers (this point is critical to get much of the > benefit of the testing, a sh script can always be committed first then > converted to a Dejagnu harness later to take advantage of all the support > Dejagnu has for testing cross-compilers)? If the format and where to put the ".sum" files is documented, that should be easy to add. > Does your test setup support additionally "ordinary" testsuite directories > for gradually accumulating tests in the style of > gcc.c-torture/{compile,execute} (test compilation / execution works with > multiple optimization options) or gcc.dg (test that error/warning messages > are as expected)? There are many such tests that Kenner has posted to the > lists over the years (for back-end bugs shown up by Ada testcases) that > should be gathered into the testsuite. At -O0 the current ACATS run is done in 45 minutes on a P3 1GHz. We can easily add a loop over compile options, but at some point there will be some thinking about what we want to require for patch and what is optional in the GCC testsuites. ACT also has a small set of tests which are not proprietary code and could be included, adding a loop over them in the ACATS script is easy. I'll go back in the list archives and add them over the time. Laurent
Subject: Re: Ada has no test suite. On Wed, 22 Oct 2003, laurent at guerby dot net wrote: > Joel Sherrill did run my original script with a cross-compiler > on a simulator and IIRC he had trouble running dejagnu (much > more complicated to set up). For dejagnu we have instructions (simtest-howto.html) that enable someone with a major patch to test (so needing to test it on three platforms) to run simulator tests without needing to understand setting up Dejagnu for cross targets. Does this Ada harness need another column for that table, or something more complicated? > At -O0 the current ACATS run is done in 45 minutes on a P3 1GHz. We can > easily add a loop over compile options, but at some point > there will be some thinking about what we want to require for patch > and what is optional in the GCC testsuites. I don't think ACATS needs running over multiple options - but where Ada code has shown up a back end bug (as in many of the testcases Kenner has sent to the lists), running over multiple options may be more useful.
Can't you just clone gcc-dg.exp? I don't see why this is hard.
Subject: Re: Ada has no test suite. On Wed, 2003-10-22 at 21:01, zack at gcc dot gnu dot org wrote: > Can't you just clone gcc-dg.exp? I don't see why this is hard. gcc-dg.exp is written in a language I don't know, I don't know how it is called, what are the available tools, how it infers its list of test, how it understands results, how to precompile common files or handle special cases. The sh script is two-third of the size of gcc-dg.exp and does what is needed. It is understandable by just about anyone able to write a sh script. It's a one line change to test post install, and another line change to test in a simulator, a few lines to loop over flags if needed. Sorry to look stupid, gcc-dg.exp is *hard* for me. As I said, if someone for whom it is "easy" to make a dg Ada framework contributes it, I'll use it (I assume there will be minimal comments on how to add a test or something to be precompiler). Sincerely, Laurent
Sorry, I didn't realize you were unfamiliar with expect in general. I suggest you check in an initial cut with the shell script you've got. Some kind of test suite is better than no test suite at all. And someone else will probably then be motivated to convert it to dejagnu.
Thanks for the last suggestion, I believe this will help moving forward, rather than trying to get a "perfect" solution and get stuck. I'm ready to put the current version, although it may be better to wait for the 3.4 branch I believe. I'd rather not push too many changes after stage 3, or that will drive Mark crazy :-) Not mentioning the fact that being a release manager myself, I certainly agree that we should avoid any possible disruption during stage 3 and concentrate on stabilizing. Arno
I'm not Mark, but I think adding a test suite for Ada should cause minimal disruption -- certainly far lower risk than the merge you just did. I also think it would be good to have the test suite included on the 3.4 branch if at all possible; that makes it much easier to check whether the Ada front end on the branch works at all, which is very useful from a release engineering standpoint.
Subject: Re: Ada has no test suite. > I'm not Mark, but I think adding a test suite for Ada should cause minimal > disruption -- certainly far lower risk than the merge you just did. That's fine with me. Assuming I have the approval of Mark, I'll add the Ada test suite. > I also think it would be good to have the test suite included on the 3.4 > branch if at all possible; that makes it much easier to check whether the > Ada front end on the branch works at all, which is very useful from a > release engineering standpoint. That's right. Arno
Subject: Re: Ada has no test suite. > > I'm not Mark, but I think adding a test suite for Ada should cause minimal > > disruption -- certainly far lower risk than the merge you just did. > > That's fine with me. > > Assuming I have the approval of Mark, I'll add the Ada test suite. It's OK with me. Thanks, -- Mark
Subject: Bug 5909 CVSROOT: /cvs/gcc Module name: gcc Changes by: charlet@gcc.gnu.org 2003-10-27 11:33:00 Modified files: gcc/testsuite : ChangeLog Added files: gcc/testsuite : README.ada Log message: PR ada/5909: * README.ada, ada/acats: Import ACATS 2.5 for GCC Ada test suite. Patches: http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/testsuite/README.ada.diff?cvsroot=gcc&r1=NONE&r2=1.1 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/testsuite/ChangeLog.diff?cvsroot=gcc&r1=1.3147&r2=1.3148
Subject: Bug 5909 CVSROOT: /cvs/gcc Module name: gcc Changes by: charlet@gcc.gnu.org 2003-10-27 14:27:17 Modified files: gcc/ada : ChangeLog exp_ch3.adb exp_ch5.adb sem_prag.adb make.adb prj-env.adb prj-env.ads g-dirope.adb Makefile.generic sem_ch12.adb sem_ch8.adb sem_ch8.ads Make-lang.in Log message: * Makefile.generic: Add missing substitution on object_deps handling. PR ada/5909: * Make-lang.in (check-ada): Enable ACATS test suite. * exp_ch3.adb: (Freeze_Array_Type): We do not need an initialization routine for types derived from String or Wide_String. They should be treated the same as String and Wide_String themselves. This caused problems with the use of Initialize_Scalars. * exp_ch5.adb: (Expand_Assign_Record): Do component-wise assignment of non-byte aligned composites. This allows use of component clauses that are not byte aligned. * sem_prag.adb: (Analyze_Pragma, case Pack): Generate warning and ignore pack if there is an attempt to pack an array of atomic objects. * make.adb, prj-env.adb, prj-env.ads: Minor reformatting * g-dirope.adb: (Basename): Check for drive letters in a pathname only on DOS based OS. * make.adb: (Gnatmake): When unable to change dir to the object dir, display the content of the parent dir of the obj dir, to try to understand why this happens. * Make-lang.in: Makefile automatically updated * sem_ch12.adb: (Inline_Instance_Body): Indicate that the save/restore of use_clauses should not be done in Save/Restore_Scope_Stack, because it is performed locally. * sem_ch8.adb: (Save_Scope_Stack, Restore_Scope_Stack): Add parameter to indicate whether use clauses should be removed/restored. * sem_ch8.ads: (Save_Scope_Stack, Restore_Scope_Stack): Add parameter to indicate whether use clauses should be removed/restored. Patches: http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ada/ChangeLog.diff?cvsroot=gcc&r1=1.363&r2=1.364 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ada/exp_ch3.adb.diff?cvsroot=gcc&r1=1.9&r2=1.10 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ada/exp_ch5.adb.diff?cvsroot=gcc&r1=1.9&r2=1.10 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ada/sem_prag.adb.diff?cvsroot=gcc&r1=1.10&r2=1.11 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ada/make.adb.diff?cvsroot=gcc&r1=1.18&r2=1.19 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ada/prj-env.adb.diff?cvsroot=gcc&r1=1.11&r2=1.12 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ada/prj-env.ads.diff?cvsroot=gcc&r1=1.9&r2=1.10 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ada/g-dirope.adb.diff?cvsroot=gcc&r1=1.11&r2=1.12 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ada/Makefile.generic.diff?cvsroot=gcc&r1=1.2&r2=1.3 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ada/sem_ch12.adb.diff?cvsroot=gcc&r1=1.23&r2=1.24 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ada/sem_ch8.adb.diff?cvsroot=gcc&r1=1.9&r2=1.10 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ada/sem_ch8.ads.diff?cvsroot=gcc&r1=1.7&r2=1.8 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ada/Make-lang.in.diff?cvsroot=gcc&r1=1.44&r2=1.45
Subject: Bug 5909 CVSROOT: /cvs/gcc Module name: gcc Changes by: charlet@gcc.gnu.org 2003-10-27 14:51:27 Modified files: gcc : ChangeLog gcc/doc : sourcebuild.texi Log message: PR ada/5909: * doc/sourcebuild.texi: Document Ada test suite. Patches: http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/ChangeLog.diff?cvsroot=gcc&r1=2.1573&r2=2.1574 http://gcc.gnu.org/cgi-bin/cvsweb.cgi/gcc/gcc/doc/sourcebuild.texi.diff?cvsroot=gcc&r1=1.32&r2=1.33
Ada now has a test suite, so I am closing this PR. Feel free to open new PR for specific improvements/suggestions, that go beyond the general issue of having an Ada test suite. Arno