Ada policy

[Mike Stump]

On Aug 30, 2004, at 2:21 PM, Richard Kenner wrote:
> second that nearly all of the test cases are proprietary code.

Yes, and this was true of quite a few of the C/C++ testcases too...  
but, this doesn't relieve the obligation to add a test case.

Don't start fixing a testcase until after you've trimmed the testcase 
down, then you will find that all bugs you fix will have cut down test 
cases.  A few bug fixes will never have testcases, that is fine.  A 
good example is a memory smasher that randomly goes in and out of 
workingness as the testcase is changed.  These will not usually have a 
testcase.  However the vast majority of the rest can have a minimal 
testcase produced.  A few will need 12 hours to cut down the testcase, 
the bulk of them should be substantially easier.

Sometimes there will be areas where the testing infrastructure doesn't 
support testing, for which you'll just skip adding tests, this too is 
fine to a degree.  In C/C++, this is done with stuff like cross module 
or cross shared library issues...  Not perfect, but the best we can do 
at the moment.  In time, we'll have a driver even for that.

We can describe the procedure to transform a proprietary testcase in 
detail into one that isn't, if you want.  There is an algorithm.  
Roughly, get testcase, remove every function, variable, block, 
statement, argument that can be that doesn't affect the testcase.  
Understand why what's left is minimal as you fix the bug, then try and 
recode the testcase to make the things that happen to be true, to be 
true.  For example, need register pressure, insert canonical code to 
increase pressure...  Mostly works for me.  It is greedy, and has near 
linear cost.  Further, at times, people automate it in various ways 
(delta, slice)...  If it is a hardship, pick an automation to handle 
the grunt work, or an intern...  Even a language agnostic one that just 
runs the algorithm line by line might be enough for you...  One that is 
done change all variable/function/type names and then examine code to 
be sure that no trade secrets or original structure or original intent, 
use or meaning is left that the customer would care about, if in doubt, 
just ask them directly.

So, I guess I am confused as to, do you just not want to do the work to 
produce the testcases, or do you lack the understanding of how to do 
it?

As for the past, I'd say that while it would be nice to get more of the 
Ada testcase that you have into the the FSF testsuite, I think it is 
reasonable to merely require this of _new_ bugfixes to new 
bugs/regressions.