RFC: Change rules for adding/changing test cases

[DJ Delorie]

> If you want to mandate something, mandate that the testcase be
> checked in simultaneously with the work that fixes it,

Many test cases are checked in by the person who discovers the bug,
long before it gets fixed.  We don't want to discourage people from
creating testcases, do we?

Plus my proposal allows for the times when a test case triggers a new
bug, for which a note could be added to the testcase.

> and that the changelog entry for the work mentions the testcase by
> name, the target on which it was verified,

IIRC the purpose of ChangeLogs is to note what changed, not why.  At
least, that's the result of the semi-annual ChangeLog flame wars.

> and mentions the bug id in the database.

There isn't always a bug for each test case.

> Mandate that the testcase be run once before the fix, and verified
> that it shows a failure, and once after the fix and that it shows a
> pass.

This doesn't cover the times when the testcase is added before a fix
is found.  It also isn't very helpful for backend-related bugs,
because often the ChangeLog - and even the code change - aren't useful
for grokking what actually causes the bug.  Knowing what causes the
bug, rather than what fixed the bug, is the key to re-fixing it in a
new backend.

> Having this pointer is better and more useful.

Not if there is no bug PR.  Not if the network is down.  Not if I'm
using a frozen source tree from some random Linux distribution such
that the testcases no longer correspond to the bug system du jour.
The information should be in CVS along with everything else.  I'm not
talking about a wealth of information here, but perhaps just a
sentence or two that point people in the right direction when they
need to figure out what's going wrong.

> The Changelog entry should cover some of the above.

The many ChangeLog wars indicate otherwise.  And if the bug is fixed
at a different time than the testcase is added, the ChangeLog is
useless anyway.

> Without a compelling benefit,

The benefit is that if I'm doing a new port and see a failure, the
first thing I look at is the test case source, and if there was a
comment there saying what it was testing and possibly how to fix it if
it fails, it would save me a boatload of time.  Even something as
simple as "If this testcase is failing, you've probably defined FOO
wrong" would be immensely useful.

> I don't think we should force this on people, as it would tend to
> discourage testcase creation, which would be bad.

What, adding one sentence is discouraging?  That's about as much
effort as goes into a Changelog entry, and we mandate those.  IMHO
it's well worth it to make people put in the effort to write down a
little bit of history for others to learn from.

> The PR can contain unlimited verbiage about the testcase.

Not if there is no PR.

> The changelog stiches the testcases to the PRs, thus linking the code,
> the documentation in the code, the verbiage in the PR, and the
> testcase in the testsuite.

The documentation in the code does not document history.  Past
discussions have made this clear; the documentation in the code exists
to describe what the *current* code does, not what it used to do.  And
often, an old test case might trigger a bug in a new section of the
code.  Having a note in there about how the new bug was fixed would be
useful too.