[RFT/RFA] Per-directory libjava builds (was RFC)

[Paolo Bonzini <paolo dot bonzini at lu dot unisi dot ch>]

Richard Henderson wrote:

> On Tue, May 10, 2005 at 11:44:58AM +0200, Paolo Bonzini wrote:
>  
>
> > The only thing to do is to avoid extra long command lines when making 
> > Makefile.defs.
> >    
> Which is actually a show-stopper.
>  
Do you mean you could not run it on Linux?  I could (and on Darwin as 
well), but anyway here is a revised patch that fixes that problem.   Now 
that the fix for PR21436 is in, it is also easier for people to test it.

Notes:

1) write_entries_to_file is very slow, but I need it to avoid extra long 
command lines.  The speed of a single invocation of 
write_entries_to_file compares with the savings in the "executing 
depfiles commands" phase of config.status (we have a lot of echo's to 
files.tmplist, instead of an echo for each .java file's dependency 
file).  Preparing Makefile.deps takes only a few seconds more than with 
my earlier patch.

2) I'm using sed instead of multiple invocations of 
write_entries_to_file.  I did not find a portable way to put a multiline 
`s' command in the Makefile, so I'm creating a temporary sed script.

3) I don't like using "x := $(shell cat abc)" instead of putting the 
assignment into an include file, but it is hard to persuade Automake not 
to move the -include statements near the very end of the file.  This 
idiom was also present in the previous patch.

4) It looks like $(shell) is expanded before executing *all* the 
commands in a rule.  write_entries_to_file uses $(shell), and that's why 
I had to write

%.list:
       @: $(shell $(mkinstalldirs) $(basename $@))
       @: $(call write_entries_to_file,$?,$@)

rather than

%.list:
       $(mkinstalldirs) $(basename $@)
       @: $(call write_entries_to_file,$?,$@)

5) This patch removes --enable-libgcj-multifile because Ranjit Mathew 
says it does not optimize anything.  That's probably because it does not 
build .lo files from multiple Java files -- only .class files.  It is 
probably possible to leave the option, but the implementation would be 
different from what I have now, so I'd prefer to reinstate it in a 
follow-up patch.

I tried to make the ChangeLog very thorough, so that it's easier to see 
what's going on.  Bootstrapped/regtested i686-pc-linux-gnu, C/C++/Java.  
Ok for mainline?

Paolo

2005-05-11  Paolo Bonzini  <bonzini@gnu.org>

	* configure.ac (--enable-libgcj-multifile): Remove.

	* Makefile.am (all_native_compiled_source_files,
	(all_native_compiled_dirs, all_java_dirs, all_java_lo_files,
	all_libgcj_lo_files, all_xlib_lo_files,
	all_java_deps_files, all_java_stamp_files,
	all_java_filelist_files): New variables.
	(EXTRA_DIST): New Automake variable.
	(CLEANFILES): Add .stamp, .list and .lo files.
	(%.list, %.lo, %.stamp, Makefile.deps): New rules.
	(classes.stamp): Depend on $(all_java_stamp_files).
	(libgcj.la): Pass $(libgcj_la_LIBADD) to the linker without
	going through write_entries_to_file.

	(all_java_source_files, all_property_files): Move earlier.
	(libgcj_la_OBJECTS, javao_files, xlib_javao_files,
	all_java_class_files): Remove.
	(gtk_awt_peer_sources): Rename to...
	(gtk_awt_peer_source_files): ... this.  Adjust throughout.

	(ONESTEP): Remove.
	(libgcj0_convenience_la_SOURCES): Remove Java source files.
	(libgcj0_convenience_la_LIBADD): Add $(all_libgcj_lo_files).
	(libgcj0_convenience_la_DEPENDENCIES): Likewise.
	(libgcj_la_SOURCES): Remove.
	(lib_gnu_awt_xlib_la_SOURCES): Remove Java source files.
	(lib_gnu_awt_xlib_la_LIBADD): Add $(all_xlib_lo_files).
	(lib_gnu_awt_xlib_la_DEPENDENCIES): Likewise.

	* configure: Regenerate.
	* Makefile.in: Regenerate.
	* external/Makefile.in: Regenerate.
	* external/sax/Makefile.in: Regenerate.
	* external/w3c_dom/Makefile.in: Regenerate.
	* gcj/Makefile.in: Regenerate.
	* include/Makefile.in: Regenerate.
	* libltdl/Makefile.in: Regenerate.
	* testsuite/Makefile.in: Regenerate.

Index: configure.ac
===================================================================
RCS file: /cvs/gcc/gcc/libjava/configure.ac,v
retrieving revision 1.28
diff -p -u -u -r1.28 configure.ac
--- configure.ac	29 Apr 2005 22:09:43 -0000	1.28
+++ configure.ac	11 May 2005 09:01:31 -0000
@@ -202,17 +202,6 @@ if test -z "$enable_hash_synchronization
    enable_hash_synchronization=$enable_hash_synchronization_default
 fi
 
-# Do we allow intermodule optimizations (i.e. compiling many files at once)?
-AC_ARG_ENABLE(libgcj-multifile,
-  AS_HELP_STRING([--enable-libgcj-multifile]
-                 [allow compilation of several files at once]),
-[case "${enableval}" in
-  yes) enable_libgcj_multifile=yes ;;
-  no)  enable_libgcj_multifile=no ;;
-  *) AC_MSG_ERROR(bad value ${enableval} for --enable-libgcj-multifile) ;;
-esac],[enable_libgcj_multifile=no])
-AM_CONDITIONAL(ONESTEP, test "$enable_libgcj_multifile" = yes)
-
 AC_ARG_WITH(java-home,
   AS_HELP_STRING([--with-java-home=DIRECTORY],
                  [value of java.home system property]),
Index: Makefile.am
===================================================================
RCS file: /cvs/gcc/gcc/libjava/Makefile.am,v
retrieving revision 1.483
diff -p -u -u -r1.483 Makefile.am
--- Makefile.am	6 May 2005 23:05:56 -0000	1.483
+++ Makefile.am	11 May 2005 09:01:32 -0000
@@ -228,6 +228,63 @@ BOOTCLASSPATH = $(here)'$(CLASSPATH_SEPA
 
 ## ################################################################
 
+# Files for which we build .lo files
+all_native_compiled_source_files = \
+    $(java_source_files) \
+    $(built_java_source_files) \
+    $(xlib_java_source_files)
+
+# Files for which we build .class files
+# It *must* begin with $(all_native_compiled_source_files)
+all_java_source_files = \
+    $(all_native_compiled_source_files) \
+    $(gtk_awt_peer_source_files) \
+    $(gnu_xml_source_files) \
+    $(javax_imageio_source_files) \
+    $(javax_xml_source_files) \
+    $(gnu_java_beans_source_files) \
+    gnu/gcj/tools/gcj_dbtool/Main.java
+
+EXTRA_DIST = $(all_java_source_files)
+
+# Build a list of native and bytecode-compiled directories.  The Makefile
+# depends on them, because they affect the definition of two variables
+# below.
+# Each directory's filelist file depends on the corresponding Java source files
+# Each class file depends on the its directory's .stamp file
+# Below, one .lo file is built per directory when the stamp file changes
+
+Makefile.deps: $(srcdir)/Makefile.in
+	@: $(call write_entries_to_file,$(all_java_source_files),files.tmplist)
+	sed 's,/[^/]*$$,,' \
+	  files.tmplist | sort | uniq > bytecodes.list
+	sed -n '1,$(words $(all_native_compiled_source_files))s,/[^/]*$$,,p' \
+	  files.tmplist | sort | uniq > native.list
+	echo 's,^\(.*\)/\([^/]*\)\.java$$,\1.list: \0\' > tmpsed
+	echo '\1/\2.class: \1.stamp,' >> tmpsed
+	sed -ftmpsed files.tmplist \
+	  > Makefile.deps
+	rm files.tmplist tmpsed
+
+all_native_compiled_dirs := $(shell test -f native.list && cat native.list)
+all_java_dirs := $(shell test -f bytecodes.list && cat bytecodes.list)
+all_java_lo_files = $(all_native_compiled_dirs:=.lo)
+all_java_deps_files = $(all_java_dirs:=.deps)
+all_java_stamp_files = $(all_java_dirs:=.stamp)
+all_java_filelist_files = $(all_java_dirs:=.list)
+
+# In the future these may be defined manually, if we decide to split libgcj.so
+# into multiple shared libraries.  For now, everything but xlib goes into
+# all_libgcj_lo_files
+all_xlib_lo_files = gnu/gcj/xlib.lo gnu/awt/xlib.lo
+all_libgcj_lo_files = $(filter-out $(all_xlib_lo_files), $(all_java_lo_files))
+
+# Include automatically generated dependencies between Java source files.
+-include Makefile.deps $(all_java_deps_files)
+
+all_property_files = $(property_files) \
+    java/util/logging/logging.properties
+
 ##
 ## How to build libgcj.a and libgcj.jar
 ##
@@ -246,23 +303,22 @@ libgij_la_LDFLAGS = -rpath $(toolexeclib
 # subsections in the linker, such as Alpha and MIPS.
 libgcj0_convenience_la_SOURCES = prims.cc jni.cc exception.cc stacktrace.cc \
 	link.cc defineclass.cc interpret.cc verify.cc \
-	$(nat_source_files) $(math_c_source_files) $(java_source_files) \
-	$(built_java_source_files) \
+	$(nat_source_files) $(math_c_source_files) \
 	$(BOEHMGC_SRC) $(NOGC_SRC) \
 	$(BACKTRACE_SRC) \
 	$(POSIX_PLATFORM_SRC) $(WIN32_PLATFORM_SRC) $(ECOS_PLATFORM_SRC) \
 	$(DARWIN_CRT_SRC) \
 	$(POSIX_THREAD_SRC) $(WIN32_THREAD_SRC) $(NO_THREAD_SRC)
 libgcj0_convenience_la_LIBADD = \
+	$(all_libgcj_lo_files) \
 	gnu-xml.lo javax-imageio.lo \
 	javax-xml.lo gnu-java-beans.lo \
 	external/sax/libsax-gcj.la \
 	external/w3c_dom/libw3c-gcj.la
+libgcj0_convenience_la_DEPENDENCIES = $(libgcj0_convenience_la_LIBADD)
 
 noinst_LTLIBRARIES = libgcj0_convenience.la
 
-libgcj_la_SOURCES =
-
 ## Objects from C++ sources in subdirs.
 nat_files = $(nat_source_files:.cc=.lo)
 xlib_nat_files = $(xlib_nat_source_files:.cc=.lo)
@@ -344,7 +400,7 @@ jni/gtk-peer/gtk_jawt.c \
 jni/classpath/native_state.c
 
 ## Java sources for Gtk peers.
-gtk_awt_peer_sources = \
+gtk_awt_peer_source_files = \
 gnu/java/awt/peer/gtk/GdkTextLayout.java \
 gnu/java/awt/peer/gtk/GdkFontMetrics.java \
 gnu/java/awt/peer/gtk/GdkFontPeer.java \
@@ -393,7 +449,7 @@ gnu/java/awt/peer/gtk/GThreadMutex.java 
 gnu/java/awt/peer/gtk/GThreadNativeMethodRunner.java
 
 
-gtk_jni_headers = $(patsubst %.java,jniinclude/%.h,$(subst /,_,$(gtk_awt_peer_sources)))
+gtk_jni_headers = $(patsubst %.java,jniinclude/%.h,$(subst /,_,$(gtk_awt_peer_source_files)))
 
 jniinclude/gnu_java_awt_peer_gtk_GdkGraphics.h: gnu/java/awt/peer/gtk/GdkGraphics.java
 jniinclude/gnu_java_awt_peer_gtk_GdkGraphics2D.h: gnu/java/awt/peer/gtk/GdkGraphics2D.java
@@ -510,11 +566,10 @@ libjawt_la_LDFLAGS = \
 	$(LIBGCJ_LD_SYMBOLIC)
 libjawt_la_LINK = $(LIBLINK)
 
-lib_gnu_awt_xlib_la_SOURCES = \
-	$(xlib_java_source_files) \
-	$(xlib_nat_source_files)
-lib_gnu_awt_xlib_la_DEPENDENCIES = libgcj-$(gcc_version).jar libgcj.la libgcj.spec
-lib_gnu_awt_xlib_la_LIBADD = $(xlib_javao_files)
+lib_gnu_awt_xlib_la_SOURCES = $(xlib_nat_source_files)
+lib_gnu_awt_xlib_la_LIBADD = $(all_xlib_lo_files)
+lib_gnu_awt_xlib_la_DEPENDENCIES = libgcj-$(gcc_version).jar libgcj.la libgcj.spec \
+	$(lib_gnu_awt_xlib_la_LIBADD)
 ## We require libstdc++-v3 to be in the same build tree.
 lib_gnu_awt_xlib_la_CPPFLAGS = \
 	$(AM_CPPFLAGS) \
@@ -528,21 +583,6 @@ lib_gnu_awt_xlib_la_LDFLAGS = ../libstdc
         -version-info `grep -v '^\#' $(srcdir)/libtool-version` $(LIBGCJ_LD_SYMBOLIC)
 lib_gnu_awt_xlib_la_LINK = $(LIBLINK)
 
-all_java_source_files = \
-    $(java_source_files) \
-    $(built_java_source_files) \
-    $(gtk_awt_peer_sources) \
-    $(xlib_java_source_files) \
-    $(gnu_xml_source_files) \
-    $(javax_imageio_source_files) \
-    $(javax_xml_source_files) \
-    $(gnu_java_beans_source_files)
-
-all_java_class_files = $(all_java_source_files:.java=.class)
-
-all_property_files = $(property_files) \
-    java/util/logging/logging.properties
-
 ## Build property files into the library.
 property_files = \
 gnu/regexp/MessagesBundle.properties \
@@ -554,32 +594,24 @@ propertyo_files = $(property_files:.prop
 %.properties.lo: %.properties
 	$(LTGCJCOMPILE) -o $@ -c $< -Wc,--resource,`echo $@ | sed "s/\.lo$$//"`
 
-if ONESTEP
+# Compile one directory in one go.  If the files have to be compiled with the
+# binary compatibility ABI, only make the .class here -- the .lo is made
+# elsewhere in this Makefile.
+
+%.list:
+	@: $(shell $(mkinstalldirs) $(basename $@))
+	@: $(call write_entries_to_file,$?,$@)
+
+$(all_java_lo_files): %.lo: %.stamp
+	$(LTGCJCOMPILE) -o $@ -c @${<:.stamp=.list}
+
+$(all_java_stamp_files): %.stamp: %.list
+	$(JAVAC) $(JCFLAGS) -classpath '' -bootclasspath $(BOOTCLASSPATH) -d $(here) \
+		-MD -MF ${@:.stamp=.deps} @$<
+	echo timestamp > $@
 
-# Compile all classfiles in one go.
-
-classes.stamp: $(all_java_source_files) gnu/gcj/tools/gcj_dbtool/Main.java
-	@echo Compiling Java sourcefiles...
-	@: $(call write_entries_to_file,$?,libgcj.sourcelist)
-	$(JAVAC) $(JCFLAGS) -classpath '' -bootclasspath $(BOOTCLASSPATH) -d $(here) @libgcj.sourcelist
-	echo > classes.stamp
-# This next rule seems backward, but reflects the fact that 1) all
-# classfiles are compiled in one go when classes.stamp is built and 2)
-# anything which depends on a particular .class file must wait until
-# this file is built.
-$(all_java_class_files) gnu/gcj/tools/gcj_dbtool/Main.class: classes.stamp
-
-else # !ONESTEP
-
-# Compile each classfile individually.
-
-.java.class:
-	$(JAVAC) $(JCFLAGS) -classpath '' -bootclasspath $(BOOTCLASSPATH) -d $(here) $<
-
-classes.stamp: $(all_java_class_files) $(all_property_files)
-	echo > classes.stamp
-
-endif
+classes.stamp: $(all_property_files) $(all_java_stamp_files)
+	echo timestamp > classes.stamp
 
 libgcj-$(gcc_version).jar: classes.stamp
 	-@rm -f libgcj-$(gcc_version).jar
@@ -605,7 +637,9 @@ libgcj-$(gcc_version).jar: classes.stamp
 	  $(ZIP) -ufM0E@ $@; \
 	done
 
-CLEANFILES = libgcj-$(gcc_version).jar classes.stamp
+CLEANFILES = libgcj-$(gcc_version).jar classes.stamp $(all_java_stamp_files) \
+	$(all_java_deps_files) $(all_java_filelist_files)
+DISTCLEANFILES = Makefile.deps native.dirs bytecompile.dirs
 
 mostlyclean-local:
 ## Use libtool rm to remove each libtool object
@@ -628,16 +662,10 @@ clean-nat:
 
 SUFFIXES = .class .java .h .properties
 
-$(javao_files) $(xlib_javao_files): %.lo: %.java
-	$(LTGCJCOMPILE) -o $@ -c $<
-
 ## Pass the list of object files to libtool in a temporary file to
 ## avoid tripping platform command line length limits.
-libgcj.la: $(libgcj_la_OBJECTS) $(libgcj_la_DEPENDENCIES)
-	@echo Creating list of files to link...
-	@: $(call write_entries_to_file,$(libgcj_la_OBJECTS),libgcj.objectlist)
-	$(libgcj_la_LINK) -objectlist libgcj.objectlist \
-	$(libgcj_la_LIBADD) \
+libgcj.la: $(libgcj_la_DEPENDENCIES)
+	$(libgcj_la_LINK) $(libgcj_la_LIBADD) \
 	-rpath $(toolexeclibdir) $(libgcj_la_LDFLAGS) $(LIBS)
 
 lib-gnu-awt-xlib.la: $(lib_gnu_awt_xlib_la_OBJECTS) $(lib_gnu_awt_xlib_la_DEPENDENCIES)
@@ -700,7 +728,7 @@ gnu-java-beans.lo: $(gnu_java_beans_sour
 
 ## Depend on the sources, even though we are going to compile the
 ## classes.
-gtk-awt-peer.lo: $(gtk_awt_peer_sources)
+gtk-awt-peer.lo: $(gtk_awt_peer_source_files)
 ## FIXME: this is ugly.  We want to make sure the .class files have
 ## been built, but we don't want a real dependency on them as this
 ## would cause our target to be rebuilt whenever any .java file is