RFC: Monitoring old PRs, new dg directives

[Marek Polacek]

On Wed, Aug 05, 2020 at 08:58:40AM +0100, Richard Sandiford wrote:
> Marek Polacek via Gcc-patches <gcc-patches@gcc.gnu.org> writes:
> > On Thu, Jul 30, 2020 at 11:54:23AM +0200, Jakub Jelinek via Gcc-patches wrote:
> >> On Tue, Jul 28, 2020 at 05:44:47PM -0400, Marek Polacek via Gcc-patches wrote:
> >> > We will still have a surfeit of bugs that we've given short shrift to, but
> >> > let's at least automate what we can.  The initial addition of the relevant
> >> > old(-ish) tests won't of course happen automagically, but it's a price I'm
> >> > willing to pay.  My goal here isn't merely to reduce the number of open PRs;
> >> > it is to improve the testing of the compiler overall.
> >> > 
> >> > Thoughts?
> >> 
> >> Looks useful to me, but I'd think it might be desirable to use separate
> >> directories for those tests, so that it is more obvious that it is a
> >> different category of tests.  Now that we use git, just using git mv
> >> to move them to another place once they are fixed for good (together with
> >> some dg-* directive tweaks) wouldn't be that much work later.
> >> 
> >> So having gcc.dg/unfixed/ , g++.dg/unfixed/ , c-c++-common/unfixed/
> >> and their torture/ suffixed variants (or better directory name for those)?
> >
> > Thanks.  I was afraid that it would cause too much friction when you happen
> > to fix one of the unfixed tests: you will have to find the correct directory
> > to put the test in and perhaps even rename the test to avoid conflicts with
> > tests with the same name in the final destination.  But it's also true that
> > git is much better at moving files, and the extra clarity might be worth the
> > occasional hassle.  It would also make it easy to skip testing unfixed tests.
> > dg-ice tests are easy to spot/grep for, but accepts-invalid/rejects-valid are
> > a different story.
> 
> FWIW, I agree with Mike that it might be better to put tests in the
> location that they'd have normally.  Some reasons:
> 
> - As already mentioned, it'd give more stable names.  That's not an
>   issue for things like git log, but it does mean that comparing one
>   set of test results with another gives a meaningful XFAIL->PASS
>   transition rather than a “removed test” + “added test” transition.
>   It's also more convenient when comparing different branches.
> 
> - There are a lot of specialised .exp harnesses, and I don't think it
>   would be a good idea to encourage all of them to have unfixed/
>   variants, or unfixed/ subdirectories.  E.g. vect.exp is not something
>   we'd want to duplicate or subclass, but there are others.
> 
> - There's a temptation to remove empty directories/harnesses when
>   they have no associated tests, so we might oscillate between having
>   unfixed/ directories and not.  (The alternative is for the directory
>   or harness to stick around based on the fact that it was useful at
>   some point in the past.)
> 
> - It's inconsistent with existing tests that are already XFAILed for all
>   targets.  Having a separate directory for completely-XFAILed tests
>   might create a requirement (or the impression of a requirement) to
>   move tests to an unfixed/ directory when XFAILing them.
> 
> - IMO it draws an artificial distinction between tests that are
>   completely XFAILed on all targets and tests that are either partly
>   XFAILed for all targets, or XFAILed only for some targets.

Those are all great points.  I have to reconsider my position once again ;-).

Perhaps we should add an // UNFIXED comment then to easily disambiguate
between tests that are not fixed.  But I fear that people won't always
use this consistently.

Marek