false regressions (was Re: results for egcs-2.91.53 586-pc-linux-gnulibc1)

[Joe Buck <jbuck at synopsys dot com>]

I wrote:
> > So what should be done?  It would be nice to have a separate status,
> > [ FATAL, XFATAL ... ]
 
mrs writes:
> No!  Please, I bed of you, don't ask for that.

It may be possible to get the information out in another way, by having
the return status of the compiler be, in itself, a test.  Then an extra
PASS would be added for each compile where the compiler does not abort.

The reason for doing that is so that we can mark a test XFAIL, but
still report a FAIL if we do not expect the compiler to abort (but only
miscompile the code).

A strict diff of .sum files may still show false regressions, but if
the cause is a compiler abort this will show up in the diff.

At this point, this is probably needed only for g++ (where compiler
aborts are still too common).

To show how this would look: my run of the test suite on egcs-1.0.3a
(sparc) has the following in g++.sum:

FAIL: g++.robertl/eb125.C illegal code (test for errors, line 6)
FAIL: g++.robertl/eb125.C semicolon missing (test for errors, line 15)
XPASS: g++.robertl/eb125.C - test is declared (test for bogus messages, line 20)
FAIL: g++.robertl/eb125.C (test for excess errors)

Note that there was an internal compiler error.  You can't tell that
from g++.sum, and a simple diff makes this look like a regression
(at least at the time the snapshot failed eb125: it had four FAIL
lines.  This one is now fixed, I think).

I would propose to add

FAIL: g++.robertl/eb125.C (test for internal compiler error)

Now, if I did the diff between this g++.sum and the one for, say,
egcs19980707 I would see something like

< XPASS: g++.robertl/eb125.C - test is declared (test for bogus messages, line 20)
> FAIL: g++.robertl/eb125.C - test is declared (test for bogus messages, line 20)

< FAIL: g++.robertl/eb125.C (test for internal compiler error)

> PASS: g++.robertl/eb125.C (test for internal compiler error)

and I could tell just from this diff that the old compiler dumps core,
without generating bogus messages, and the new compiler does not dump
core, but generates bogus messages.