Re: [PATCH] Disable guality tests for powerpc*-linux*

[Bill Schmidt <wschmidt at linux dot vnet dot ibm dot com>]

Hi Jakub,

On Tue, 2016-03-29 at 08:53 +0200, Jakub Jelinek wrote:
> On Mon, Mar 28, 2016 at 07:38:46PM -0500, Bill Schmidt wrote:
> > For a long time we've had hundreds of failing guality tests.  These
> > failures don't seem to have any correlation with gdb functionality for
> > POWER, which is working fine.  At this point the value of these tests to
> > us seems questionable.  Fixing these is such low priority that it is
> > unlikely we will ever get around to it.  In the meanwhile, the failures
> > simply clutter up our regression test reports.  Thus I'd like to disable
> > them, and that's what this test does.
> > 
> > Verified to remove hundreds of failure messages on
> > powerpc64le-unknown-linux-gnu. :)  Is this ok for trunk?
> 
> This is IMNSHO very wrong, you then lose tracking of regressions in the
> debug info quality.  It is true that the debug info quality is already
> pretty bad  on powerpc*, it would be really very much desirable if
> anyone had time to analyze some of them and improve stuff,
> but we at least shouldn't regress.  Guality testsuite has various FAILs
> and/or XFAILs on lots of architectures, the problem is that the testing
> matrix is simply too large to have them in the testcases
> - it depends on the target, various ISA settings on the target, on the
> optimization level (most of the guality tests are torture tested through
> -O0 up to -O3 with extra flags), and in some cases also on the version of
> the used GDB.
> 
> For guality, the most effective test for regressions is simply always
> running contrib/test_summary after all your bootstraps and then just
> diffing up that against the same from earlier bootstrap.

And of course we do this, and we can keep doing it.  My main purpose in
opening this issue is to try to understand whether we are getting any
benefit from these tests, rather than just noise.

When you say that "the debug info quality is already pretty bad on
powerpc*," do you mean that it is known to be bad, or simply that we
have a lot of guality failures that may or may not indicate that the
debug info is bad?  I don't have experiential evidence of bad debug info
that shows up during debugging sessions.  Perhaps these are corner cases
that I will never encounter in practice?  Or perhaps the tests are just
badly formed?

The failing tests have all been bit-rotten (or never worked) since
before I joined this project, and from what others tell me, for at least
a decade.  As you suggest here, others have always told me just to
ignore the existing guality failures.  However, this can easily lead to
a culture of "ignore any guality failure, that stuff is junk" which can
cause regressions to be missed.  (I can't say that I've actually
observed this, but it is a concern I have.)

I have been consistently told that the same situation exists on most of
the supported targets, again because of the size of the testing matrix.
I'd be interested in knowing if this is true, or just anecdotal.

The other point, "it would be really very much desirable if
anyone had time to analyze some of them and improve stuff," has to be
answered by "apparently nobody does."  I am currently tracking well over
200 improvements I would like to see made to the powerpc64le target
alone.  Investigating old guality failures isn't even on that list.  Our
team won't have time for it, and if we have bounty money to spend, it
will be spent on more important things.  That's just the economic
reality, not a desire to disrespect the guality tests or anyone
associated with them.

>From my limited perspective, it seems like the guality tests are unique
within the test suite as a set of tests that everyone just expects to
have lots of failures.  Is that healthy?  Will it ever change?

That said, it is clear that you feel the guality tests provide at least
some value in their present state, so we can continue to live with
things as they are.  I'm just curious how others feel about the state of
these tests.

Thanks,
Bill

> 
> 	Jakub
>