Re: Power of the G++ test suite

[Robert Lipe <robertl at dgii dot com>]

[ re: user frustration over submitted tests not being committed ] 

Joe Buck wrote:

> Generally speaking, the test suite is being maintained as a regression
> test suite.  That is, tests are added along with fixes: e.g. a patch
> will be checked in along with a test case, which fails before the patch,
> and passed afterwards.

I'm not the maintainer of anything important in this regard, but I would
like to see anything that's inarguably broken flagged with a testcase.
There has been some (non-list) discussion about whether new testcases
that aren't accompanied by a fix should be marked XFAIL or not.  I'd
have to look back over the thread, but I think I recall Jeff and Jason
disagreeing.  Perhaps we should consider adding a classification that
indicates that something is known to be broken and just hasn't been
fixed yet.  This way people don't wig out when they see N new testcase
failures because they don't realize that they see this not becuase the
quality dropped but because there were N new testcases added.

> In addition, some tests have been added to mark important failures or
> published bug reports (the ones Ropert Lipe contributed).  So these
> cases are more like what you contributed, I think.

Those very tests have taught me (and I'm not singling Carlos or anyone
else out) that the majority of the bug reports on egcs-bugs are just
not bothering with.  In the last batch, I picked through some 3500
messages and wound up reducing them to something like 130 entries that
could reasonably be considered as testcases.  Martin and Jason shredded
another few dozen of them and I think that we wound up with something
around a two dozen actual bugs that needed fixing.  Right now, I have
another 5100 messages that I'm wading through.  I'll confess that I'm
cherry-picking the ICEs first; they're inarguably bad.

Since I don't do C++, I'm not comfortable committing ad-hoc testcases
since it often generates work for the Jasons and Martins to clean them
up or identify them as bogus. I was leaving them for the appropriate
maintainer to comment on.

> Oh, please don't take it so personally.  Even the most skilled
> developers will have patches rejected or ignored once in a while, and

There was some discussion given to feeding the lists into a PRMS-style
tracking system so that proposed patches such as new testcase entries
could be assigned to an individual or group for consideration.  I have
no idea about the status of it.  If it's awaiting human resources,
perhaps a Call For Volunteers could be issued...

RJL