testing framework not catching --enable-checking "crashes"?

[craig at jcb-sc dot com]

Learning about, and exploring, the DejaGnu stuff, in conjunction with
trying to create a proper test case for a minor bug-fix (which I just
submitted to egcs-patches), I'm noticing some evidence that at least
some tests "pass" despite "crashing" due to the --enable-checking
mechanism being triggered.

For example, here's some of the output of a runtest with
`--all --verbose --verbose' of the noncompile.exp script:

--------
dest in standard_file is deer
doing compile
Invoking the compiler as /home3/craig/gnu/f77-e/gcc/xgcc -B/home3/craig/gnu/f77-e/gcc/ /home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c  -c  -o /home3/craig/gnu/f77-e/gcc/testsuite/990416-1.o
Executing on host: /home3/craig/gnu/f77-e/gcc/xgcc -B/home3/craig/gnu/f77-e/gcc/ /home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c  -c  -o /home3/craig/gnu/f77-e/gcc/testsuite/990416-1.o   
pid is 16933 -16933
waitres is 16933 7 0 1
close result is 16933 7 0 1
output is /home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:4: parse error before `frame_state'
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c: In function `copy_reg':
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: `reg' undeclared (first use in this function)
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: (Each undeclared identifier is reported only once
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: for each function it appears in.)
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: `udata' undeclared (first use in this function)
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: warning: initialization makes pointer from integer without a cast
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:7: `target_udata' undeclared (first use in this function)
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:7: warning: initialization makes pointer from integer without a cast
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:9: ../../g77-e/gcc/expr.c:10806: Expect 't', have 'error_mark'


Checking pattern "sparc-*-sunos*" with i686-pc-linux-gnu
Checking pattern "alpha*-*-*" with i686-pc-linux-gnu
compiler exited with status 1
output is:
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:4: parse error before `frame_state'
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c: In function `copy_reg':
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: `reg' undeclared (first use in this function)
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: (Each undeclared identifier is reported only once
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: for each function it appears in.)
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: `udata' undeclared (first use in this function)
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: warning: initialization makes pointer from integer without a cast
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:7: `target_udata' undeclared (first use in this function)
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:7: warning: initialization makes pointer from integer without a cast
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:9: ../../g77-e/gcc/expr.c:10806: Expect 't', have 'error_mark'


Checking pattern "sparc-*-sunos*" with i686-pc-linux-gnu
Checking pattern "alpha*-*-*" with i686-pc-linux-gnu

Checking:
.*:4:.*:6:.*:6:.*:6:.*:6:.*:6:.*:7:.*:7:
to see if it matches:
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:4: parse error before `frame_state'
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c: In function `copy_reg':
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: `reg' undeclared (first use in this function)
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: (Each undeclared identifier is reported only once
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: for each function it appears in.)
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: `udata' undeclared (first use in this function)
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:6: warning: initialization makes pointer from integer without a cast
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:7: `target_udata' undeclared (first use in this function)
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:7: warning: initialization makes pointer from integer without a cast
/home3/craig/gnu/g77-e/gcc/testsuite/gcc.c-torture/noncompile/990416-1.c:9: ../../g77-e/gcc/expr.c:10806: Expect 't', have 'error_mark'


Yes, it matches.

PASS: gcc.c-torture/noncompile/990416-1.c
--------

Should the pattern(s) for each test to match be adjusted so that the
"Expect..." message causes no match?  Or perhaps the --enable-checking
mechanism should trigger a call to abort() to force the tests to
fail (assuming that'd do the trick)?  I don't understand this stuff
nearly well enough to guess at which, if either, is right, or even
be sure that the above represents a real problem.

The reason the above test caught my special attention is that my proposed
patch *seemed* to fix it, based on "make check-gcc" runs, but really
did not, and I don't understand why that particular test was claimed
by runtest to fail in one run, but later to pass, and I cannot reproduce
the failure (despite temporarily reversing my patch to the gcc front end).

I mention that in case it turns out the above behavior is a bug, and
might even be some kind of Heisenbug.

        tq vm, (burley)