[Bug testsuite/23149] New: gcov test assertion naming

["jsm28 at gcc dot gnu dot org" <gcc-bugzilla at gcc dot gnu dot org>]

On 20050728 the results in gcc.sum for gcov-4.c said:

PASS: gcc.misc-tests/gcov-4.c (test for excess errors)
PASS: gcc.misc-tests/gcov-4.c execution test
PASS: gcc.misc-tests/gcov-4.c gcov

while on 20050730 with no change to the test they said:

PASS: gcc.misc-tests/gcov-4.c (test for excess errors)
PASS: gcc.misc-tests/gcov-4.c execution test
FAIL: 96:is -:should be 1
FAIL: gcc.misc-tests/gcov-4.c gcov: 1 failures in line counts, 0 in branch
percentages, 0 in return percentages

The text which appears after PASS or FAIL is meant to be a name for a test
assertion which is constant as long as the testcase itself is unchanged.  Thus a
testsuite regression should be indicated by PASS: <test name> changing to FAIL:
<test name> rather than by the removal of a PASS and its replacement by FAILs
with different names; new FAILs in general can indicate newly added tests which
fail from the start and may or may not be regressions.  (contrib/compare_tests
illustrates this logic, showing these new failures less helpfully than if the
PASS and FAIL lines had consistent test names, though it's not the tool I'm
actually using for this comparison.)

Thus, each gcov output line which is checked should have its own fixed test
assertion name which appears after PASS if the output is correct and after FAIL
if the output is incorrect; the .log file might have additional information
about what the difference was, but not on the PASS or FAIL lines.  The final
summary PASS or FAIL line probably isn't needed in these circumstances, but if
it is again the test assertion name should not depend on whether the test passes
or fails (information about the number and kind of failures is a separate
matter, not for the PASS or FAIL lines).

-- 
           Summary: gcov test assertion naming
           Product: gcc
           Version: 4.1.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P2
         Component: testsuite
        AssignedTo: unassigned at gcc dot gnu dot org
        ReportedBy: jsm28 at gcc dot gnu dot org
                CC: gcc-bugs at gcc dot gnu dot org


http://gcc.gnu.org/bugzilla/show_bug.cgi?id=23149