Re: [PATCH] Temporarily enable ac_fold_checking for --enable-checking=yes.

[Jeffrey Law <law at redhat dot com>]

On Sat, 2007-06-23 at 12:54 -0600, Mark Mitchell wrote:

> I love automated testing.
Likewise.

> 
> But, I'm concerned about some of the things that we're doing.  We have a
> few testcases in the C testsuite that take up huge fractions of the
> total time.  The libstdc++ testsuite now runs for a long time after the
> G++ testsuite is done, even running both in parallel.  Additional
> testing during bootstraps makes things slower.
Yup.  And if you happen to also run any of the proprietary testsuites,
then the testing aspect gets to be borderline crazy.  If you happen to
work on a multilib target, then, well, testing has been taking more than
24 hours for years :(

> 
> I have two concerns.  First, I think some of the testing we do is not
> delivering value proportional to cost.  Is 20061226-1.c (sp?) really
> delivering anything close to the same percent of our quality that it
> costs in testing? 
Extremely valid concerns.  Many years ago there was talk of trying to
somehow break the GCC regression suite into different buckets of tests.
We were mostly looking at functional boundaries to help with evaluating
the status of new ports.  That effort never got off the ground, but we
could look at similar strategies to make our testing more time
effective.

The trick (of course) is finding the right tests to move into the 
"expensive" bucket that, perhaps, we don't run for every checkin,
but let the auto-testers run.

Other approaches might look for a slick way to cut down on the system
overhead necessary to run a single testfile -- while 20061226-1 takes a
lot of time, we also burn an insane amount of time on all the small
tests with the multiple shell, compiler, assembler, linker invocations. 
We've got some hacks to cut down (don't run the loop related iterations
if there are no loops, don't re-run the executable if it was unchanged
when the new options, etc).  But there's no systematic approach to 
avoiding useless work.  I'm really not even sure where to start with
this angle, but last I looked with oprofile, we burned an inordinate
amount of time firing up shells, fork/exec, gcc 8 or more times per
torture test...  God help you if you happened to use a system with a
slow process startup (hpux, solaris).

the other long term thing I want to see would (of course) be unit
testing.  Pie in the sky, but for dealing with some classes of
problems I suspect it'd be far more efficient than our current
approach.

Jeff