Re: ACATS legal status cleared by FSF

[dewar at gnat dot com]

Note that just *reading* B tests to see if the output is correct is a
very difficult task, one that only someone with quite a bit of ACVC/ACATS
validation experience can do. A formal validation run using these tests
often involves several days of painstaking manual work by someone who
is an expert in the B tests to assure compliance.

Consider various possibilities:

1. You must not make changes to the compiler that change the B tests baselines.

     This is far too restrictive, it would forbid even fixing a spelling error.
     In practice fixing a bug in one part of the compiler can often change
     an error message elsewhere in one of the B tests (by implementing
     better error recovery). Most often such changes are for the better.

2. If the B test baseline changes, verify the baseline change and adjust the
   baseline.

     This would be fine, except that the verification process typically
     requires someone with very good expertise in Ada semantics, AND
     very good familiarity with the ACATS test suite. There are not
     many such people in the world.

A huge effort from any Ada vendor goes into making sure that the compiler
passes all the B tests. Many have actively questioned the value of this
effort in the past, but if you want to formally validate you have no 
choice. But this does not necessarily mean that these are well chosen
tests from the point of view of our use here at gnu.org.