Re: ACATS legal status cleared by FSF

[dewar at gnat dot com]

<<BTW is there any record of the existing noncompile testsuite catching
problems, or did it just prevent any serious error message work by
scaring people?
>>

Well sure, there have been some cases in which a subtle change to the
front end caused an error message to be lost, but it is infrequent. 
And for sure it has not stopped ACT from doing serious error message
work, which has always been a focus for us (getting the best possible
error messages), but we often often have cases where we make a simple
improvement in an error message, and 90% of the work is checking very
carefully through the B-test baselines to ensure that the baselines
should indeed be updated and nothing has slipped by.

It is certainly not possible to provide a comprehensive test suite for
use with the gcc tree. That's because the most valuable tests from
the test suites in use are the tests in the Compaq and ACT test suites
None of the Compaq suite (called "DEC test suite" informally) are
available for use, due to licensing restrictions, and the great majority
of the ACT tests are not available, since they are proprietary customer
code.

So what we are doing at the gcc site is to put a subset of high value
tests that are worth the effort running. What I am saying is that the
C tests meet this criterion, but the B tests don't.

Yes occasionally, a change that someone makes to the system will break a
B test, so what? Much more often it will be the case if people make changes
to the front end that they break one of the tests in the DEC test suite
or ACT test suite. In either case, we here at ACT have to figure out how
to repair the problem, and I would guess that the B tests issues will play
a very minor role.

If someone sees a misspelling in an error message, I am happy for them to
just fix it, and do not want to inhibit such a change just because of the
effort of updating the B tests. Indeed if someone does update the B test
baseline, it would be more work for us to check that they had done this
update correctly than to do it ourselves, and that careful check would be
required in any case. Remember that the B test baseline is an artifact that
is maintained not for testing purposes primarily, but for validation 
purposes, something we are not interested in for the gcc version per se.

Now of course there are changes to error messages that require a huge
amount of work in all test suites. A good example is an enhancement
request we have logged that suggests a nicer treatment of continuation
messages, so that a multi-line message is obviously a multi-line messagre
rather than separate messages. This is a fairly simple patch that could
be done in half an hour, but the consequences to the base lines of all
test suites would be ferocious, so that is why this change is still on the
list (there are lots of other things on the list). In fact perhaps we can
publish at least some of our list of suggested enhancements so that people
can try to do some of them :-)

I actually think that by far the most valuable addition to the C tests would
be to add some of the tests from the ACT test suite that ACT wrote, and that
are therefore potentially available. As soon as we have the tree issues
fully under control (most notably the docuemntation is still a real issue),
we will send some of these tests along.

Robert Dewar

And P.S. we are certainly not suggesting hiding the B tests, we are just
suggesting not worrying about them too much :-)