Re: ACATS legal status cleared by FSF

[Zack Weinberg <zack at codesourcery dot com>]

On Thu, Dec 06, 2001 at 05:12:14PM -0500, Geert Bosch wrote:
> 
> It is virtually impossible for people to "break" these tests, which
> is why I say they are of no value. Even if people *do* manage to
> break them (in the hypothetical case that the maintainers would not
> catch the error before approving), this will not go unnoticed for a
> long time anyway. In the mean time, the *only* programs affected are
> programs with fatal errors to start with.
> 
> I would be surprised if there would even be consensus to run just
> the executable ACATS tests as part of make check, since this would
> already double testing time for all contributors. Adding testing
> requirements is not free, and there needs to be a benefit to it.
> 
> For most of the executable ACATS tests I think there is a good
> benefit/cost ratio for the front end, and even for the back end. For
> that reason I am happy to see Laurent doing the work to get them
> integrated. For the B tests, adding testing is of near-zero value at
> a high cost.
> 
> Zack, I'd like to see very good reasons why you think it is
> reasonable to significantly increase of required volunteer time to
> make GNAT changes and that way hinder development and maintenance.

I'm 100% confident that there is value to having "make check" drive at
least some set of Ada tests.  Ever since I've been a member of the
project I've been seeing patches to the back end go by with a note
"this code is only used by Ada" or "test case is in Ada" (with the
implication that writing a C testcase is impossible or at least too
much work).  How much back end logic is that, that the current test
suite doesn't even touch?

It may make sense not to run the ACATS B tests by default, but they
should at least be _present_ in the repository so that everyone is on
an equal footing for changes that affect error messages.

zw