Re: Power of the G++ test suite

[Jeffrey A Law <law at cygnus dot com>]

  In message <19980722223459.D7103@dgii.com>you write:
  > I'm not the maintainer of anything important in this regard, but I would
  > like to see anything that's inarguably broken flagged with a testcase.
  > There has been some (non-list) discussion about whether new testcases
  > that aren't accompanied by a fix should be marked XFAIL or not.  I'd
  > have to look back over the thread, but I think I recall Jeff and Jason
  > disagreeing.
Jason and I may be disagreeing due to differences in the kinds of tests
added to the two testsuites.

g++ tests are typically for front-end issues.

gcc tests are typically for code generation issues.

Because the g++ tests are testing for front-end bugs, they typically
do not benefit from being run multiple times with different options.

However, the gcc tests do benefit greatly from multiple testruns, and
as a result of running multiple tests, they can be very difficult to
xfail appropriately (without generating a bunch of XPASS messages).

Thus the two tests have some fundamentally different goals.  As such we
may (or may not) want to treat them in a different manner.

  > Perhaps we should consider adding a classification that
  > indicates that something is known to be broken and just hasn't been
  > fixed yet.  This way people don't wig out when they see N new testcase
  > failures because they don't realize that they see this not becuase the
  > quality dropped but because there were N new testcases added.
One of the ideas that has come up internally at Cygnus is the concept of
marking tests as "might fail".

The result of running a test that "might fail" is either a PASS or XFAIL.

The downside is few folks ever look at XFAILed tests, so the bugs may
never get fixed.  Thus we may only want to do this for some subset of 
older tests, leaving new ones as failures to encourage folks to fix
the bugs :-)

The other thing that we've been talking about inside Cygnus is to
somehow mark some tests as more important than others, or at least
breaking them down into more manageable groups.  For example group
all the label value tests together, all the stdarg/varargs tests,
complex values, structure returns, etc etc.

This might be of more value to folks that write lots of ports and need
to be able to classify groups of failing tests in some kind of priority
order.  (ie, stuff like trampolines is usually one of the last things
I work on when doing a port, so it would be nice to be able to ignore
all the tramopline tests early in the port development cycle).

We're certainly open to more thoughts on the general topic of how to
improve the testsuites and testing process.



  > Those very tests have taught me (and I'm not singling Carlos or anyone
  > else out) that the majority of the bug reports on egcs-bugs are just
  > not bothering with.  In the last batch, I picked through some 3500
  > messages and wound up reducing them to something like 130 entries that
  > could reasonably be considered as testcases.  Martin and Jason shredded
  > another few dozen of them and I think that we wound up with something
  > around a two dozen actual bugs that needed fixing.  Right now, I have
  > another 5100 messages that I'm wading through.  I'll confess that I'm
  > cherry-picking the ICEs first; they're inarguably bad.
Right.  I found similar stuff when I created g++.law many years ago
and I was working from messages that had already been sorted out!
(the old g++ buglist distributions for those who remember them).



  > There was some discussion given to feeding the lists into a PRMS-style
  > tracking system so that proposed patches such as new testcase entries
  > could be assigned to an individual or group for consideration.  I have
  > no idea about the status of it.  If it's awaiting human resources,
  > perhaps a Call For Volunteers could be issued...
We do have someone working on the PRMS database stuff; he'd been held
up for a couple months due to lameness on the Cygnus sysadmin side.
There's still issues to deal with, but it has started moving forward
again.



jeff