This is the mail archive of the
gcc@gcc.gnu.org
mailing list for the GCC project.
Re: GCC Buildbot Update - Definition of regression
On Wed, 11 Oct 2017, Martin Sebor wrote:
> I don't have a strong opinion on the definition of a Regression
> in this context but I would very much like to see status changes
> highlighted in the test results to indicate that something that
There are lots of things that are useful *if* you have someone actively
reviewing them for every test run (maybe several times a day) and alerting
people / filing bugs in Bugzilla if there are problems. Some of those
things, however, are likely to have too many false positives for a display
people can quickly look at to see if the build is red or green, or for
automatically telling people their patch broke something.
If we can clean up results for each system the bot runs tests on -
XFAILing and filing bugs in Bugzilla for failures where there isn't a
reasonably simple and obvious fix - we can make green mean "no FAILs or
ERRORs" (remembering the possibility that with very broken testing,
sometimes an ERROR might only be in the .log not the .sum). Other
differences (such as PASS -> UNSUPPORTED) can then be reviewed manually by
someone who takes responsibility for doing so, resulting in bugs being
filed if appropriate, without affecting the basic red/green status.
(Variants such as green meaning "no FAILs or ERRORs, except for failing
guality tests where there should be no regressions" are possible as well,
for cases like that where PASS/FAIL status depends on non-GCC components
and meaningfully selective XFAILing is hard.)
--
Joseph S. Myers
joseph@codesourcery.com