Re: RFC: Change rules for adding/changing test cases

[law at redhat dot com]

 In message <200203200127.g2K1R3200849@greed.delorie.com>, DJ Delorie writes:
 > > If you want to mandate something, mandate that the testcase be
 > > checked in simultaneously with the work that fixes it,
 > 
 > Many test cases are checked in by the person who discovers the bug,
 > long before it gets fixed.  We don't want to discourage people from
 > creating testcases, do we?
Agreed.  From time to time I check in testcases without fixes so that the
testcase doesn't get lost.

 > > and that the changelog entry for the work mentions the testcase by
 > > name, the target on which it was verified,
 > 
 > IIRC the purpose of ChangeLogs is to note what changed, not why.  At
 > least, that's the result of the semi-annual ChangeLog flame wars.
Right.  ChangeLog documents what changed, not why.  The why belongs as 
comments in the code.

In the case of a test, knowing what problem it was meant to trigger would
be awful helpful.  And the most natural place for that is in the test itself.

 > > and mentions the bug id in the database.
 > 
 > There isn't always a bug for each test case.
Even if there were, it's still quite valuable to have a basic explanation of
what the test is meant to check in the test itself.  The long lengthy analysis
certainly belongs in the bug database, but knowing that test X is meant to
test for a problem with pass X doing incorrect action Y is still quite 
valuable.

 > This doesn't cover the times when the testcase is added before a fix
 > is found.  It also isn't very helpful for backend-related bugs,
 > because often the ChangeLog - and even the code change - aren't useful
 > for grokking what actually causes the bug.  Knowing what causes the
 > bug, rather than what fixed the bug, is the key to re-fixing it in a
 > new backend.
Right.

 > Not if there is no bug PR.  Not if the network is down.  Not if I'm
 > using a frozen source tree from some random Linux distribution such
 > that the testcases no longer correspond to the bug system du jour.
 > The information should be in CVS along with everything else.  I'm not
 > talking about a wealth of information here, but perhaps just a
 > sentence or two that point people in the right direction when they
 > need to figure out what's going wrong.
Right.  Something like:

/* The loop optimizer incorrectly hoists expression p + q out of the 
   main loop because the CFG has become inaccurate.  */

Would be awful helpful from time to time.

 > The many ChangeLog wars indicate otherwise.  And if the bug is fixed
 > at a different time than the testcase is added, the ChangeLog is
 > useless anyway.
And why in the world would I search through all the ChangeLogs to know what
a test is supposed to do when it most naturally belongs in the test itself?


 > The benefit is that if I'm doing a new port and see a failure, the
 > first thing I look at is the test case source, and if there was a
 > comment there saying what it was testing and possibly how to fix it if
 > it fails, it would save me a boatload of time.  Even something as
 > simple as "If this testcase is failing, you've probably defined FOO
 > wrong" would be immensely useful.
Right.

jeff