Pushing XFAILed test cases

[Martin Sebor]

On 7/16/21 9:32 AM, Thomas Schwinge wrote:
> [Also including <gcc@gcc.gnu.org> for guidance.]
> 
> 
> Hi!
> 
> (I'm not involved in or familiar with Sandra's Fortran TS29113 work, just
> commenting generally here.)
> 
> 
> On 2021-07-16T09:52:28+0200, Thomas Koenig via Gcc-patches <gcc-patches@gcc.gnu.org> wrote:
>> It is my understanding that it is not gcc policy to add xfailed test
>> cases for things that do not yet work. Rather, xfail is for tests that
>> later turn out not to work, especially on certain architectures.
> 
> That's not current practice, as far as I can tell.  I'm certainly
> "guilty" of pushing lots of XFAILed test cases (or, most often,
> individual XFAILed DejaGnu directives), and I see a good number of others
> GCC folks do that, too.  Ideally with but casually also without
> corresponding GCC PRs filed.  If without, then of course should have
> suitable commentary inside the test case file.  Time span of addressing
> the XFAILs ranging between days and years.
> 
> In my opinion, if a test case has been written and analyzed, why
> shouldn't you push it, even if (parts of) it don't quite work yet?  (If
> someone -- at another time, possibly -- then implements the missing
> functionality/fixes the bugs, the XFAILs turn into XPASSes, thus serving
> to demonstrate the effect of code changes.
> 
> Otherwise -- and I've run into that just yesterday... -- effort spent on
> such test cases simply gets lost "in the noise of the mailing list
> archives", until re-discovered, or -- in my case -- re-implemented and
> then re-discovered by chance.
> 
> We nowadays even have a way to mark up ICEing test cases ('dg-ice'),
> which has been used to push test cases that ICE for '{ target *-*-* }'.
> 
> 
> Of course, we shall assume a certain level of quality in the XFAILed test
> cases: I'm certainly not suggesting we put any random junk into the
> testsuite, coarsely XFAILed.  (I have not reviewed Sandra's test cases to
> that effect, but knowing here, I'd be surprised if that were the problem
> here.)
> 
> 
> Not trying to overrule you, just sharing my opinion -- now happy to hear
> others.  :-)

I've also been xfailing individual directives in new tests, with
or without PRs tracking the corresponding limitations (not so much
outright bugs as future enhancements).  The practice has been
discussed in the past and (IIRC) there was general agreement with
it.  Marek even formalized some of it for the C++ front end by
adding support for one or more dg- directives (I think dg-ice was
one of them). The discussion I recall is here:

https://gcc.gnu.org/pipermail/gcc-patches/2020-July/550913.html

Martin