This is the mail archive of the gcc@gcc.gnu.org mailing list for the GCC project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

[PATCH] RE: gcc parallel make check


> The splits are in the Makefiles, see check_gcc_parallelize

attached is a patch to improve the parallel performance of 'make -jXX -k check-fortran'. For XX=16, this yields ~50% speedup, and even with XX=4 we still have 15%, the measured slowdown at XX=1 (<2%) is in the noise of testing. The patch is a simple update of the 'check_gfortran_parallelize' variable, updating it from its 2008 values to a set that I found +- optimal based on several tests. Detailed timings are :

# timings/trunk-check-fortran
#cores      average    std. dev. #tests
     1      2955.32        75.06      3
     2      1735.30       122.26      3
     4       929.51        54.19      3
     8       470.29         7.85      3
    16       468.09         4.29      3
    32       466.06         1.24      3

# timings/patched-check-fortran
#cores      average    std. dev. #tests
     1      3008.89        16.38      3
     2      1534.17       118.33      3
     4       800.18        31.71      3
     8       418.71         0.20      2
    16       298.29         5.86      3
    32       299.84         1.34      3

There is no effect on a full 'make -j32 -k check' as other goals run for much longer (to be looked at in a followup).

A second part of the patch is a new file 'contrib/generate_tcl_patterns.sh' which generates the needed regexp to do the split based on an input of the files in the target directory. It basically groups the initial characters such that each regexp tries not to exceed a maximum number of files. So, the number of files is used as a proxy for the runtime. While I don't feel to strong about adding this (shell/gawk) script, it certainly is convenient, and makes sure that no characters are missing from the regexp. The maximum number of files per regexp is an input, testing (-j16) with 200, 300, 400 I found that 300 was optimal for testsuite/gfortran.dg, but this will depend on many things. 

A sample run would look like

gcc/gcc/testsuite/gfortran.dg> ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/"
Adding label:  p matching files:499
Adding label:  c matching files:497
Adding label:  a matching files:448
Adding label:  i matching files:350
Adding label:  d matching files:245
Adding label:  s matching files:211
Adding label:  b matching files:206
Adding label:  t matching files:180
Adding label:  f matching files:173
Adding label:  e matching files:166
Adding label:  r matching files:165
Adding label:  n matching files:162
Adding label:  mu matching files:278
Adding label:  wlgo matching files:284
Adding label:  vhzPkqWx_-9876543210ZYXVUTSRQONMLKJIHGFEDCBAyj matching files:94
patterns:
dg.exp=gfortran.dg/p* \
dg.exp=gfortran.dg/c* \
dg.exp=gfortran.dg/a* \
dg.exp=gfortran.dg/i* \
dg.exp=gfortran.dg/\[wlgo\]* \
dg.exp=gfortran.dg/\[mu\]* \
dg.exp=gfortran.dg/d* \
dg.exp=gfortran.dg/s* \
dg.exp=gfortran.dg/b* \
dg.exp=gfortran.dg/t* \
dg.exp=gfortran.dg/f* \
dg.exp=gfortran.dg/e* \
dg.exp=gfortran.dg/r* \
dg.exp=gfortran.dg/n* \
dg.exp=gfortran.dg/\[vhzPkqWx_-9876543210ZYXVUTSRQONMLKJIHGFEDCBAyj\]* \

Is the current attached patch OK for trunk ?

contrib/ChangeLog

2014-09-05  Joost VandeVondele  <vondele@gcc.gnu.org>

       * generate_tcl_patterns.sh: New file.

gcc/fortran/ChangeLog

 2014-09-05  Joost VandeVondele  <vondele@gcc.gnu.org>

       * Make-lang.in (check_gfortran_parallelize): improved parallelism.


Index: contrib/generate_tcl_patterns.sh
===================================================================
--- contrib/generate_tcl_patterns.sh	(revision 0)
+++ contrib/generate_tcl_patterns.sh	(revision 0)
@@ -0,0 +1,86 @@
+#! /bin/sh
+
+#
+# based on a list of filenames as input,
+# generate regexps that match subsets trying to not exceed a
+# 'maxcount' parameter. Most useful to generate the
+# check_LANG_parallelize assignments needed to split
+# testsuite directories, defining prefix appropriately.
+#
+# Example usage:
+#   cd gcc/gcc/testsuite/gfortran.dg
+#   ls -1 | ../../../contrib/generate_tcl_patterns.sh 300 "dg.exp=gfortran.dg/"
+#
+# the first parameter is the maximum number of files.
+# the second parameter the prefix used for printing.
+#
+
+# Copyright (C) 2014 Free Software Foundation
+# Contributed by Joost VandeVondele <Joost.VandeVondele@mat.ethz.ch>
+#
+# This file is part of GCC.
+#
+# GCC is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 3, or (at your option)
+# any later version.
+#
+# GCC is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with GCC; see the file COPYING.  If not, write to
+# the Free Software Foundation, 51 Franklin Street, Fifth Floor,
+# Boston, MA 02110-1301, USA.
+
+gawk -v maxcount=$1 -v prefix=$2 '
+BEGIN{
+  # list of allowed starting chars for a file name in a dir to split
+  achars="abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_"
+}
+{
+  nfiles++ ; files[nfiles]=$1
+}
+END{
+  for(i=1; i<=length(achars); i++) count[substr(achars,i,1)]=0
+  for(i=1; i<=nfiles; i++) {
+     if (length(files[i]>0)) { count[substr(files[i],1,1)]++ }
+  };
+  asort(count,ordered)
+  countsingle=0
+  groups=0
+  label=""
+  for(i=length(achars);i>=1;i--) {
+    countsingle=countsingle+ordered[i] 
+    for(j=1;j<=length(achars);j++) {
+       if(count[substr(achars,j,1)]==ordered[i]) found=substr(achars,j,1)
+    }
+    count[found]=-1
+    label=label found
+    if(i==1) { val=maxcount+1 } else { val=ordered[i-1] }
+    if(countsingle+val>maxcount) {
+      subset[label]=countsingle
+      print "Adding label: ", label, "matching files:" countsingle
+      groups++
+      countsingle=0
+      label=""
+    }
+  }
+  print "patterns:"
+  asort(subset,ordered)
+  for(i=groups;i>=1;i--) {
+    for(j in subset){
+      if(subset[j]==ordered[i]) found=j
+    }
+    subset[found]=-1
+    if (length(found)==1) {
+       printf("%s%s* \\\n",prefix,found)
+    } else {
+       printf("%s\\[%s\\]* \\\n",prefix,found)
+    }
+  }
+}
+'
+
Index: gcc/fortran/Make-lang.in
===================================================================
--- gcc/fortran/Make-lang.in	(revision 214949)
+++ gcc/fortran/Make-lang.in	(working copy)
@@ -168,12 +168,22 @@ check-fortran-subtargets : check-gfortra
 lang_checks += check-gfortran
 lang_checks_parallelized += check-gfortran
 # For description see comment above check_gcc_parallelize in gcc/Makefile.in.
-check_gfortran_parallelize = dg.exp=gfortran.dg/\[adAD\]* \
-			     dg.exp=gfortran.dg/\[bcBC\]* \
-			     dg.exp=gfortran.dg/\[nopNOP\]* \
-			     dg.exp=gfortran.dg/\[isuvISUV\]* \
-			     dg.exp=gfortran.dg/\[efhkqrxzEFHKQRXZ\]* \
-			     dg.exp=gfortran.dg/\[0-9gjlmtwyGJLMTWY\]*
+check_gfortran_parallelize = execute.exp \
+			dg.exp=gfortran.dg/p* \
+			dg.exp=gfortran.dg/c* \
+			dg.exp=gfortran.dg/a* \
+			dg.exp=gfortran.dg/i* \
+			dg.exp=gfortran.dg/\[wlgo\]* \
+			dg.exp=gfortran.dg/\[mu\]* \
+			dg.exp=gfortran.dg/d* \
+			dg.exp=gfortran.dg/s* \
+			dg.exp=gfortran.dg/b* \
+			dg.exp=gfortran.dg/t* \
+			dg.exp=gfortran.dg/f* \
+			dg.exp=gfortran.dg/e* \
+			dg.exp=gfortran.dg/r* \
+			dg.exp=gfortran.dg/n* \
+			dg.exp=gfortran.dg/\[vhzPkqWx_-9876543210ZYXVUTSRQONMLKJIHGFEDCBAyj\]*
 
 # GFORTRAN documentation.
 GFORTRAN_TEXI = \

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]