RFC: Monitoring old PRs, new dg directives
Wed Aug 5 07:58:40 GMT 2020
Marek Polacek via Gcc-patches <email@example.com> writes:
> On Thu, Jul 30, 2020 at 11:54:23AM +0200, Jakub Jelinek via Gcc-patches wrote:
>> On Tue, Jul 28, 2020 at 05:44:47PM -0400, Marek Polacek via Gcc-patches wrote:
>> > We will still have a surfeit of bugs that we've given short shrift to, but
>> > let's at least automate what we can. The initial addition of the relevant
>> > old(-ish) tests won't of course happen automagically, but it's a price I'm
>> > willing to pay. My goal here isn't merely to reduce the number of open PRs;
>> > it is to improve the testing of the compiler overall.
>> > Thoughts?
>> Looks useful to me, but I'd think it might be desirable to use separate
>> directories for those tests, so that it is more obvious that it is a
>> different category of tests. Now that we use git, just using git mv
>> to move them to another place once they are fixed for good (together with
>> some dg-* directive tweaks) wouldn't be that much work later.
>> So having gcc.dg/unfixed/ , g++.dg/unfixed/ , c-c++-common/unfixed/
>> and their torture/ suffixed variants (or better directory name for those)?
> Thanks. I was afraid that it would cause too much friction when you happen
> to fix one of the unfixed tests: you will have to find the correct directory
> to put the test in and perhaps even rename the test to avoid conflicts with
> tests with the same name in the final destination. But it's also true that
> git is much better at moving files, and the extra clarity might be worth the
> occasional hassle. It would also make it easy to skip testing unfixed tests.
> dg-ice tests are easy to spot/grep for, but accepts-invalid/rejects-valid are
> a different story.
FWIW, I agree with Mike that it might be better to put tests in the
location that they'd have normally. Some reasons:
- As already mentioned, it'd give more stable names. That's not an
issue for things like git log, but it does mean that comparing one
set of test results with another gives a meaningful XFAIL->PASS
transition rather than a “removed test” + “added test” transition.
It's also more convenient when comparing different branches.
- There are a lot of specialised .exp harnesses, and I don't think it
would be a good idea to encourage all of them to have unfixed/
variants, or unfixed/ subdirectories. E.g. vect.exp is not something
we'd want to duplicate or subclass, but there are others.
- There's a temptation to remove empty directories/harnesses when
they have no associated tests, so we might oscillate between having
unfixed/ directories and not. (The alternative is for the directory
or harness to stick around based on the fact that it was useful at
some point in the past.)
- It's inconsistent with existing tests that are already XFAILed for all
targets. Having a separate directory for completely-XFAILed tests
might create a requirement (or the impression of a requirement) to
move tests to an unfixed/ directory when XFAILing them.
- IMO it draws an artificial distinction between tests that are
completely XFAILed on all targets and tests that are either partly
XFAILed for all targets, or XFAILed only for some targets.
More information about the Gcc-patches