[build, testsuite, v3] Increase gcc, g++, gfortran and libstdc++-v3 testsuite parallelism

[Rainer Orth <ro at CeBiTec dot Uni-Bielefeld dot DE>]

[Please keep me on the Cc:, I'm not subscribed to libstdc++.  Thanks.]

make check times on slow machines with many cores (like the
UltraSPARC-T1/T2) is currently limited by too little parallelism in the
testsuite.  E.g., on a Sun Fire T5220 with a single UltraSPARC-T2 CPU
(1.2 GHz, 8 cores with 8 strands/hw threads each), you can easily run
bootstrap with make -j128 and the build completes in about 2 1/2 hours.
A subsequent make -j128 -k check (both 32 and 64-bit) takes 6 hours,
while even on my laptop (Core2 Duo T7500, 2.2 GHz, 2 cores), build and
test times are roughly identical.  This is not only a SPARC problem,
though: with 8-socket 8-core Nehalem EX machines coming, we have 64
cores per machine, doubling that with HT, so gcc testing won't exploit
the available parallelism.

As a first step to improve things, I've increased parallelism in the
testsuites with the largest number of tests (using wc -l <testsuite>.sum
as an estimate):

    2361 gcc/testsuite/ada/acats/acats.sum				 1
   49546 gcc/testsuite/g++/g++.sum					20
  145926 gcc/testsuite/gcc/gcc.sum					60
   74594 gcc/testsuite/gfortran/gfortran.sum				30
    1673 gcc/testsuite/gnat/gnat.sum					 1
    4079 gcc/testsuite/objc/objc.sum					 2
    5287 i386-pc-solaris2.11/libffi/testsuite/libffi.sum		 2
    5020 i386-pc-solaris2.11/libgomp/testsuite/libgomp.sum		 2
    5196 i386-pc-solaris2.11/libjava/testsuite/libjava.sum		 2
   14841 i386-pc-solaris2.11/libstdc++-v3/testsuite/libstdc++.sum	 6
  308523 total

With 300+ ktests total and a goal of spreading them about 128 jobs, we
get 2410 tests for each, and I've used that number as a guideline.  The
third column in the table above gives the number of jobs necessary to
achieve that goal.  One could go even further here considering there are
machines like the Sun Fire T5440 with 4 US-T2 CPUs, yielding 256 cores
and thus 512 jobs, but while single-cpu T1/T2 machines are readily
available, the latter are probably too expensive and I fear that further
increasing parallelism right now might add too much overhead on small
machines.

While using the test counts as estimates worked reasonably well in most
cases, it broke down in two cases: 

* The libstdc++ tests take much more time per test, so I had to increase
  parallelism considerably beyond the first estimate.

* The gcc.c-torture/compile/limits-fnargs.c test case alone takes 16+
  minutes to compile, the test is run 8 times with different
  optimization levels and the number is doubled again by testing both
  the 32 and 64-bit multilibs, so this single test case alone takes more
  than 4 hours to complete.  This is way over the top, so I've halved
  the number of function arguments.  This way, it still takes 3+
  minutes, which is bad, but manageable.

There are still some testsuites that run excessively long (last column
is runtime in seconds):

g++23.sep             6665
libstdc++13.sep       6920
libstdc++12.sep       7191
libjava               7994
gcc16.sep             8212
libstdc++15.sep       8466
libgomp               8615

I've so far refrained from parallelizing the libjava and libgomp
testsuites runs, and some of the libstdc++ test names use very long
common prefixes.  Still, the new state is far better than the current
one, so we might incrementally improve things if libstdc++ or the serial
testsuite runs begin to further dominate the make check times.

The changes below achive this and still try to keep the patterns
manageable.  Here are the make -j<2 * ncpu> -k check times for two tests
(T5220 as above, X7350 is a Sun Fire X4450 with 4 x 2.93 GHz Xeon X7350):

CPU		mainline	parallel

X7350	-j32	 63 min		 56 min
US-T2   -j128	358 min         143 min  

As you can see, on faster CPUs the change is in the noise (or the
parallel test is actually faster), while on the T2 there's a factor of 2.5
speedup.

mail-report.log is identical between both runs.

There's one partially related change included: even before this change,
there could be a strange double summary line in mail-report.log:

                === libstdc++ Summary ===

# of expected passes            13934
# of unexpected failures        8
# of unexpected successes       2
# of expected failures          118
# of unsupported tests          732

                === libstdc++ Summary ===

I could trace this to the fact that currently no libstdc++ ABI baselines
exist for Solaris 2 (being worked on), thus runtest abi.exp produces

[...]
                === libstdc++ Summary for unix ===

Running target unix/-m64
Running /vol/gcc/src/hg/trunk/local/libstdc++-v3/testsuite/libstdc++-abi/abi.exp ...

                === libstdc++ Summary for unix/-m64 ===


                === libstdc++ Summary ===


This situation is likely to occur more often in the future when runtest
is run with a .exp file that doesn't match any tests on a given platform
(like i386.exp on SPARC).  The fix is trivial: only emit the summary
line if any of the test counts is non-zero.

Ok for mainline and the 4.4 and 4.5 branches after some soaktime?

Thanks.
	Rainer


2010-03-07  Rainer Orth  <ro@CeBiTec.Uni-Bielefeld.DE>

	contrib:
	* dg-extract-results.sh (total.awk): Only print combined summary
	for non-zero total.

	gcc:
	* Makefile.in (check_gcc_parallelize): Increase parallelism.
	(check_p_numbers): Extend to 70.

	gcc/cp:
	* Make-lang.in (check_g++_parallelize): Increase parallelism.

	gcc/fortran:
	* Make-lang.in (check_gfortran_parallelize): Increase parallelism.

	gcc/testsuite:
	* gcc.c-torture/compile/limits-fnargs.c (PAR4HALF): Define.
	(func): Replace PAR4 by PAR4HALF.
	(ARG4HALF): Define.
	(caller): Call func with ARG4HALF.

	libstdc++-v3:
	* testsuite/Makefile.am (check_DEJAGNU_normal_targets): Extend to 17.
	Increase parallelism.
	* testsuite/Makefile.in: Regenerate.

diff -r cbdbde9aae00 contrib/dg-extract-results.sh
--- a/contrib/dg-extract-results.sh	Mon Jul 19 11:13:06 2010 +0200
+++ b/contrib/dg-extract-results.sh	Tue Jul 20 20:51:39 2010 +0200
@@ -401,7 +401,7 @@
 /^# of unresolved testcases/	{ unrescnt += \$5 }
 /^# of unsupported tests/	{ unsupcnt += \$5 }
 END {
-  printf ("\n\t\t=== %s Summary ===\n\n", tool)
+  if (passcnt + failcnt + xpasscnt + xfailcnt + untstcnt + unrescnt + unsupcnt != 0) printf ("\n\t\t=== %s Summary ===\n\n", tool)
   if (passcnt != 0) printf ("# of expected passes\t\t%d\n", passcnt)
   if (failcnt != 0) printf ("# of unexpected failures\t%d\n", failcnt)
   if (xpasscnt != 0) printf ("# of unexpected successes\t%d\n", xpasscnt)
diff -r cbdbde9aae00 gcc/Makefile.in
--- a/gcc/Makefile.in	Mon Jul 19 11:13:06 2010 +0200
+++ b/gcc/Makefile.in	Tue Jul 20 20:51:39 2010 +0200
@@ -494,14 +494,75 @@
 # very long running, but still should be split of from the check-parallel-$lang
 # remaining tests runtest invocation), they should be concatenated with commas.
 # Note that [a-zA-Z] wildcards need to have []s prefixed with \ (needed
-# by tcl) and as the *.exp arguments are mached both as is and with
+# by tcl) and as the *.exp arguments are matched both as is and with
 # */ prefixed to it in runtest_file_p, it is usually desirable to include
 # a subdirectory name.
-check_gcc_parallelize=execute.exp=execute/2* \
-		      execute.exp=execute/\[013-9a-zA-Z\]* \
-		      compile.exp dg.exp \
-		      dg-torture.exp,builtins.exp \
-		      struct-layout-1.exp,unsorted.exp,stackalign.exp,i386.exp
+check_gcc_parallelize=compile.exp=compile/\[013-9\]* \
+		      compile.exp=compile/2000* \
+		      compile.exp=compile/2001* \
+		      compile.exp=compile/2002* \
+		      compile.exp=compile/2003* \
+		      compile.exp=compile/2004* \
+		      compile.exp=compile/2005* \
+		      compile.exp=compile/2006* \
+		      compile.exp=compile/2007* \
+		      compile.exp=compile/2008* \
+		      compile.exp=compile/2009* \
+		      compile.exp=compile/2010* \
+		      compile.exp=compile/201\[1-9\]* \
+		      compile.exp=compile/20\[2-9\]* \
+		      compile.exp=compile/2\[1-9\]* \
+		      compile.exp=compile/\[a-oA-O\]* \
+		      compile.exp=compile/\[pP\]* \
+		      compile.exp=compile/\[q-zQ-Z\]* \
+		      execute.exp=execute/\[013-9\]* \
+		      execute.exp=execute/2000* \
+		      execute.exp=execute/2001* \
+		      execute.exp=execute/2002* \
+		      execute.exp=execute/2003* \
+		      execute.exp=execute/2004* \
+		      execute.exp=execute/2005* \
+		      execute.exp=execute/2006* \
+		      execute.exp=execute/2007* \
+		      execute.exp=execute/2008* \
+		      execute.exp=execute/2009* \
+		      execute.exp=execute/2010* \
+		      execute.exp=execute/201\[1-9\]* \
+		      execute.exp=execute/20\[2-9\]* \
+		      execute.exp=execute/2\[1-9\]* \
+		      execute.exp=execute/\[a-eA-E\]* \
+		      execute.exp=execute/\[f-mF-M\]* \
+		      execute.exp=execute/\[n-pN-P\]* \
+		      execute.exp=execute/\[q-zQ-Z\]* \
+		      unsorted.exp=unsorted/\[0-9a-cA-C\]* \
+		      unsorted.exp=unsorted/\[dD\]* \
+		      unsorted.exp=unsorted/\[e-tE-T\]* \
+		      unsorted.exp=unsorted/\[u-zU-Z\]* \
+		      dg.exp=gcc.dg/\[0-9a-bA-B\]* \
+		      dg.exp=gcc.dg/\[cC\]* \
+		      dg.exp=gcc.dg/\[d-iD-I\]* \
+		      dg.exp=gcc.dg/\[j-nJ-N\]* \
+		      dg.exp=gcc.dg/\[o-pO-P\]* \
+		      dg.exp=gcc.dg/\[q-tQ-T\]* \
+		      dg.exp=gcc.dg/\[u-zU-Z\]* \
+		      dg.exp=c-c++-common/* \
+		      dg-torture.exp=torture/\[0-9a-bA-B\]* \
+		      dg-torture.exp=torture/\[c-oC-O\]* \
+		      dg-torture.exp=torture/\[pP\]* \
+		      dg-torture.exp=torture/\[q-zQ-Z\]* \
+		      builtins.exp \
+		      cpp.exp \
+		      debug.exp \
+		      format.exp \
+		      i386.exp \
+		      ieee.exp \
+		      noncompile.exp \
+		      pch.exp \
+		      stackalign.exp \
+		      struct-layout-1.exp \
+		      test-framework.exp \
+		      tree-ssa.exp \
+		      vect.exp
 lang_opt_files=@lang_opt_files@ $(srcdir)/c-family/c.opt $(srcdir)/common.opt
 lang_specs_files=@lang_specs_files@
 lang_tree_files=@lang_tree_files@
@@ -4892,7 +4953,13 @@
 check_p_subno=$(word 2,$(subst _, ,$*))
 check_p_comma=,
 check_p_subwork=$(subst $(check_p_comma), ,$(if $(check_p_subno),$(word $(check_p_subno),$(check_p_vars))))
-check_p_numbers=1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
+check_p_numbers=1 2 3 4 5 6 7 8 9 10 \
+	11 12 13 14 15 16 17 18 19 20 \
+	21 22 23 24 25 26 27 28 29 30 \
+	31 32 33 34 35 36 37 38 39 40 \
+	41 42 43 44 45 46 47 48 49 50 \
+	51 52 53 54 55 56 57 58 59 60 \
+	61 62 63 64 65 66 67 68 69 70
 check_p_subdir=$(subst _,,$*)
 check_p_subdirs=$(wordlist 1,$(words $(check_$*_parallelize)),$(check_p_numbers))
 
diff -r cbdbde9aae00 gcc/cp/Make-lang.in
--- a/gcc/cp/Make-lang.in	Mon Jul 19 11:13:06 2010 +0200
+++ b/gcc/cp/Make-lang.in	Tue Jul 20 20:51:39 2010 +0200
@@ -143,7 +143,29 @@
 lang_checks += check-g++
 lang_checks_parallelized += check-g++
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_g++_parallelize = old-deja.exp dg.exp
+check_g++_parallelize = old-deja.exp=g++.old-deja/g++.\[a-b\]* \
+			old-deja.exp=g++.old-deja/g++.\[c-k\]* \
+			old-deja.exp=g++.old-deja/g++.\[l-m\]* \
+			old-deja.exp=g++.old-deja/g++.\[n-o\]* \
+			old-deja.exp=g++.old-deja/g++.\[p-r\]* \
+			old-deja.exp=g++.old-deja/g++.\[s-z\]* \
+			debug.exp \
+			dg.exp=c-c++-common/* \
+			dg.exp=g++.dg/\[0-9a-bA-B\]* \
+			dg.exp=g++.dg/\[cC\]\[a-pA-P\]* \
+			dg.exp=g++.dg/\[cC\]\[q-zQ-Z0-9_\]* \
+			dg.exp=g++.dg/\[dD\]* \
+			dg.exp=g++.dg/\[eE\]* \
+			dg.exp=g++.dg/\[f-kF-K\]* \
+			dg.exp=g++.dg/\[l-nL-N\]* \
+			dg.exp=g++.dg/\[oO\]* \
+			dg.exp=g++.dg/\[pP\]* \
+			dg.exp=g++.dg/\[q-sQ-S\]* \
+			dg.exp=g++.dg/\[tT\]\[a-ea-E\]* \
+			dg.exp=g++.dg/\[tT\]\[f-oF-O\]* \
+			dg.exp=g++.dg/\[tT\]\[p-zP-Z0-9_\]* \
+			dg.exp=g++.dg/\[u-zU-Z\]* \
+			dg-torture.exp
 
 #
 # Install hooks:
diff -r cbdbde9aae00 gcc/fortran/Make-lang.in
--- a/gcc/fortran/Make-lang.in	Mon Jul 19 11:13:06 2010 +0200
+++ b/gcc/fortran/Make-lang.in	Tue Jul 20 20:51:39 2010 +0200
@@ -170,9 +170,40 @@
 lang_checks += check-gfortran
 lang_checks_parallelized += check-gfortran
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_gfortran_parallelize = dg.exp=gfortran.dg/\[a-cA-C\]* \
-			     dg.exp=gfortran.dg/\[d-mD-M\]* \
-			     dg.exp=gfortran.dg/\[n-zN-Z0-9\]*
+check_gfortran_parallelize = dg.exp=gfortran.dg/\[aA\]\[a-lA-L\]* \
+			     dg.exp=gfortran.dg/\[aA\]\[m-rM-R\]* \
+			     dg.exp=gfortran.dg/\[aA\]\[s-zS-Z0-9_\]* \
+			     dg.exp=gfortran.dg/\[bB\]\[a-nA-N\]* \
+			     dg.exp=gfortran.dg/\[bB\]\[o-zO-Z0-9_\]* \
+			     dg.exp=gfortran.dg/\[cC\]\[a-mA-M\]* \
+			     dg.exp=gfortran.dg/\[cC\]\[n-zN-Z0-9_\]* \
+			     dg.exp=gfortran.dg/\[dD\]\[a-mA-M\]* \
+			     dg.exp=gfortran.dg/\[dD\]\[n-zN-Z0-9_\]* \
+			     dg.exp=gfortran.dg/\[eE\]* \
+			     dg.exp=gfortran.dg/\[fF\]* \
+			     dg.exp=gfortran.dg/\[gG\]\[a-zA-Z\]* \
+			     dg.exp=gfortran.dg/\[gG\]\[0-9_\]* \
+			     dg.exp=gfortran.dg/\[hH\]* \
+			     dg.exp=gfortran.dg/\[iI\]\[a-mA-M\]* \
+			     dg.exp=gfortran.dg/\[iI\]\[nN\]* \
+			     dg.exp=gfortran.dg/\[iI\]\[o-zO-Z0-9_\]* \
+			     dg.exp=gfortran.dg/\[j-lJ-L\]* \
+			     dg.exp=gfortran.dg/\[mM\]* \
+			     dg.exp=gfortran.dg/\[nN\]* \
+			     dg.exp=gfortran.dg/\[oO\]* \
+			     dg.exp=gfortran.dg/\[pP\]\[a-qA-Q\]* \
+			     dg.exp=gfortran.dg/\[pP\]\[r-zR-Z0-9_\]* \
+			     dg.exp=gfortran.dg/\[q-rQ-R\]* \
+			     dg.exp=gfortran.dg/\[sS\]* \
+			     dg.exp=gfortran.dg/\[tT\]* \
+			     dg.exp=gfortran.dg/\[u-vU-V\]* \
+			     dg.exp=gfortran.dg/\[wW\]* \
+			     dg.exp=gfortran.dg/\[x-zX-Z0-9_\]* \
+			     compile.exp \
+			     execute.exp=execute/\[a-eA-E\]* \
+			     execute.exp=execute/\[f-iF-I\]* \
+			     execute.exp=execute/\[j-tJ-T\]* \
+			     execute.exp=execute/\[u-zU-Z0-9_\]*
 
 # GFORTRAN documentation.
 GFORTRAN_TEXI = \
diff -r cbdbde9aae00 gcc/testsuite/gcc.c-torture/compile/limits-fnargs.c
--- a/gcc/testsuite/gcc.c-torture/compile/limits-fnargs.c	Mon Jul 19 11:13:06 2010 +0200
+++ b/gcc/testsuite/gcc.c-torture/compile/limits-fnargs.c	Tue Jul 20 20:51:39 2010 +0200
@@ -4,19 +4,21 @@
 #define PAR2 PAR1, PAR1, PAR1, PAR1, PAR1, PAR1, PAR1, PAR1, PAR1, PAR1
 #define PAR3 PAR2, PAR2, PAR2, PAR2, PAR2, PAR2, PAR2, PAR2, PAR2, PAR2
 #define PAR4 PAR3, PAR3, PAR3, PAR3, PAR3, PAR3, PAR3, PAR3, PAR3, PAR3
+#define PAR4HALF PAR3, PAR3, PAR3, PAR3, PAR3
 #define PAR5 PAR4, PAR4, PAR4, PAR4, PAR4, PAR4, PAR4, PAR4, PAR4, PAR4
 #define PAR6 PAR5, PAR5, PAR5, PAR5, PAR5, PAR5, PAR5, PAR5, PAR5, PAR5
 
-extern void func (PAR4);
+extern void func (PAR4HALF);
 
 #define ARG1 0,1,2,3,4,5,6,7,8,9
 #define ARG2 ARG1, ARG1, ARG1, ARG1, ARG1, ARG1, ARG1, ARG1, ARG1, ARG1
 #define ARG3 ARG2, ARG2, ARG2, ARG2, ARG2, ARG2, ARG2, ARG2, ARG2, ARG2
 #define ARG4 ARG3, ARG3, ARG3, ARG3, ARG3, ARG3, ARG3, ARG3, ARG3, ARG3
+#define ARG4HALF ARG3, ARG3, ARG3, ARG3, ARG3
 #define ARG5 ARG4, ARG4, ARG4, ARG4, ARG4, ARG4, ARG4, ARG4, ARG4, ARG4
 #define ARG5HALF ARG5, ARG5, ARG5, ARG5, ARG5
 
 void caller(void)
 {
-  func (ARG4);
+  func (ARG4HALF);
 }
diff -r cbdbde9aae00 libstdc++-v3/testsuite/Makefile.am
--- a/libstdc++-v3/testsuite/Makefile.am	Mon Jul 19 11:13:06 2010 +0200
+++ b/libstdc++-v3/testsuite/Makefile.am	Tue Jul 20 20:51:39 2010 +0200
@@ -100,7 +100,7 @@
 	@test ! -f $*/site.exp || mv $*/site.exp $*/site.bak
 	@mv $*/site.exp.tmp $*/site.exp
 
-check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3)
+check_DEJAGNU_normal_targets = $(patsubst %,check-DEJAGNUnormal%,0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17)
 $(check_DEJAGNU_normal_targets): check-DEJAGNUnormal%: normal%/site.exp
 
 # Run the testsuite in normal mode.
@@ -110,8 +110,15 @@
 	if [ -z "$*$(filter-out --target_board=%, $(RUNTESTFLAGS))" ] \
 	    && [ "$(filter -j, $(MFLAGS))" = "-j" ]; then \
 	  $(MAKE) $(AM_MAKEFLAGS) check-DEJAGNUnormal0 check-DEJAGNUnormal1 \
-		  check-DEJAGNUnormal2 check-DEJAGNUnormal3; \
-	  for idx in 0 1 2 3; do \
+		  check-DEJAGNUnormal2 check-DEJAGNUnormal3 \
+		  check-DEJAGNUnormal4 check-DEJAGNUnormal5 \
+		  check-DEJAGNUnormal6 check-DEJAGNUnormal7 \
+		  check-DEJAGNUnormal8 check-DEJAGNUnormal9 \
+		  check-DEJAGNUnormal10 check-DEJAGNUnormal11 \
+		  check-DEJAGNUnormal12 check-DEJAGNUnormal13 \
+		  check-DEJAGNUnormal14 check-DEJAGNUnormal15 \
+		  check-DEJAGNUnormal16 check-DEJAGNUnormal17; \
+	  for idx in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17; do \
 	    mv -f normal$$idx/libstdc++.sum normal$$idx/libstdc++.sum.sep; \
 	    mv -f normal$$idx/libstdc++.log normal$$idx/libstdc++.log.sep; \
 	  done; \
@@ -120,11 +127,26 @@
 	  $(SHELL) $(srcdir)/../../contrib/dg-extract-results.sh \
 	    libstdc++.sum.sep normal0/libstdc++.sum.sep \
 	    normal1/libstdc++.sum.sep normal2/libstdc++.sum.sep \
-	    normal3/libstdc++.sum.sep > libstdc++.sum; \
+	    normal3/libstdc++.sum.sep normal4/libstdc++.sum.sep \
+	    normal5/libstdc++.sum.sep normal6/libstdc++.sum.sep \
+	    normal7/libstdc++.sum.sep normal8/libstdc++.sum.sep \
+	    normal9/libstdc++.sum.sep normal10/libstdc++.sum.sep \
+	    normal11/libstdc++.sum.sep normal12/libstdc++.sum.sep \
+	    normal13/libstdc++.sum.sep normal14/libstdc++.sum.sep \
+	    normal15/libstdc++.sum.sep normal16/libstdc++.sum.sep \
+	    normal17/libstdc++.sum.sep \
+	    normal14/libstdc++.sum.sep > libstdc++.sum; \
 	  $(SHELL) $(srcdir)/../../contrib/dg-extract-results.sh -L \
 	    libstdc++.log.sep normal0/libstdc++.log.sep \
 	    normal1/libstdc++.log.sep normal2/libstdc++.log.sep \
-	    normal3/libstdc++.log.sep > libstdc++.log; \
+	    normal3/libstdc++.log.sep normal4/libstdc++.log.sep \
+	    normal5/libstdc++.log.sep normal6/libstdc++.log.sep \
+	    normal7/libstdc++.log.sep normal8/libstdc++.log.sep \
+	    normal9/libstdc++.log.sep normal10/libstdc++.log.sep \
+	    normal11/libstdc++.log.sep normal12/libstdc++.log.sep \
+	    normal13/libstdc++.log.sep normal14/libstdc++.log.sep \
+	    normal15/libstdc++.log.sep normal16/libstdc++.log.sep \
+	    normal17/libstdc++.log.sep > libstdc++.log; \
 	  exit 0; \
 	fi; \
 	srcdir=`$(am__cd) $(srcdir) && pwd`; export srcdir; \
@@ -140,13 +162,41 @@
 			$(RUNTESTFLAGS) abi.exp; \
 	    else echo "WARNING: could not find \`runtest'" 1>&2; :;\
 	    fi; \
-	    dirs="`cd $$srcdir; echo [013-9][0-9]_*/* [ab]* de* [ep]*/*`";; \
+	    dirs="`cd $$srcdir; echo [01][0-9]_*/*`";; \
 	  normal1) \
-	    dirs="`cd $$srcdir; echo 2[0-2]_*/*`";; \
+	    dirs="`cd $$srcdir; echo [3-9][0-9]_*/*`";; \
 	  normal2) \
-	    dirs="`cd $$srcdir; echo 2[4-9]_*/*`";; \
+	    dirs="`cd $$srcdir; echo 20_*/*`";; \
 	  normal3) \
-	    dirs="`cd $$srcdir; echo 23_*/* t*/*`";; \
+	    dirs="`cd $$srcdir; echo 21_*/*`";; \
+	  normal4) \
+	    dirs="`cd $$srcdir; echo 22_*/*`";; \
+	  normal5) \
+	    dirs="`cd $$srcdir; echo 23_*/[0-9a-hA-H]*`";; \
+	  normal6) \
+	    dirs="`cd $$srcdir; echo 23_*/[i-mI-M]*`";; \
+	  normal7) \
+	    dirs="`cd $$srcdir; echo 23_*/[n-uN-U]*`";; \
+	  normal8) \
+	    dirs="`cd $$srcdir; echo 23_*/[v-zV-Z]*`";; \
+	  normal9) \
+	    dirs="`cd $$srcdir; echo 24_*/*`";; \
+	  normal10) \
+	    dirs="`cd $$srcdir; echo 25_*/*`";; \
+	  normal11) \
+	    dirs="`cd $$srcdir; echo 26_*/*`";; \
+	  normal12) \
+	    dirs="`cd $$srcdir; echo 27_*/*`";; \
+	  normal13) \
+	    dirs="`cd $$srcdir; echo 2[89]_*/*`";; \
+	  normal14) \
+	    dirs="`cd $$srcdir; echo [ab]* de*`";; \
+	  normal15) \
+	    dirs="`cd $$srcdir; echo e*/*`";; \
+	  normal16) \
+	    dirs="`cd $$srcdir; echo p*/*`";; \
+	  normal17) \
+	    dirs="`cd $$srcdir; echo t*/*`";; \
 	esac; \
 	if [ -n "$*" ]; then cd "$*"; fi; \
 	if $(SHELL) -c "$$runtest --version" > /dev/null 2>&1; then \

-- 
-----------------------------------------------------------------------------
Rainer Orth, Center for Biotechnology, Bielefeld University