RFC: Automatic dependency generation for libcpp

[Zack Weinberg <zack at codesourcery dot com>]

In the last discussion of automatic dependency generation for gcc,
Alexandre Oliva pointed out that it's quite easy to use Automake's
'depcomp' mechanism without the rest of Automake.  I looked into this,
and found it to be true; you do have to write your own autoconf glue,
but that's okay.

This patch is a trial balloon, which implements automatic dependency
generation in this fashion for the libcpp directory.  libcpp's needs
are much simpler than gcc's, so it makes a good test.  I have not
attempted to do anything clever with dependencies on generated
headers; the one generated-header dependency in libcpp continues to be
listed explicitly in the Makefile.

My proposal is to wait until Friday for comments on this patch, then
check it in (assuming no problems are found), wait until after the GCC
Summit for further problems to show, then implement the same logic for
the gcc subdirectory.  Again, nothing clever will be done with
dependencies on generated headers, except that I might move the
invocations of mkconfig.sh into config.status so that those headers
will exist ahead of time.

Note that, unlike Automake's scheme, there is no way to disable
generation of dependencies.  This would be fine in libcpp, assuming
the generated-header dependency remains explicit, but I do not have
confidence in the correctness of gcc's Makefile without the header
dependencies, so I want them generated always.

The patch to depcomp is tangential.  It makes depcomp work with newer
versions of HP's "acc" compiler, the ones shipped with ia64-hpux
systems: this compiler does not recognize *any* of the conventional
switches for dependency output, tho it does have its own, and
furthermore its -E output is formatted differently from the usual.  As
this version of acc is the non-GCC compiler most conveniently
available to me, I wanted to be able to test with it.  I  have tried to
send this patch to automake upstream, but my mails to automake-patches
do not get through.  If someone would relay it I would be grateful.

zw

top level:
        * depcomp: Update from automake CVS.  Add 'ia64hp' stanza.
        In 'cpp' stanza, support '#line' as well as '# '.
config:
        * depstand.m4, lead-dot.m4: New files.
libcpp:
        * configure.ac: Invoke ZW_CREATE_DEPDIR and
        ZW_PROG_COMPILER_DEPENDENCIES.
        * aclocal.m4, configure: Regenerate.
        * Makefile.in (DEPMODE, DEPDIR, depcomp, COMPILE.base, COMPILE): 
        New variables.
        (distclean): Clean up $(DEPDIR) and its contents.
        (.c.o): Use $(COMPILE).
        Include $(DEPDIR)/*.Po for most object->header dependencies.


===================================================================
Index: depcomp
--- depcomp	23 Sep 2004 01:21:50 -0000	1.3
+++ depcomp	7 Jun 2005 18:56:53 -0000
@@ -1,9 +1,9 @@
 #! /bin/sh
 # depcomp - compile a program generating dependencies as side-effects
 
-scriptversion=2004-05-31.23
+scriptversion=2005-05-16.16
 
-# Copyright (C) 1999, 2000, 2003, 2004 Free Software Foundation, Inc.
+# Copyright (C) 1999, 2000, 2003, 2004, 2005 Free Software Foundation, Inc.
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -17,8 +17,8 @@ scriptversion=2004-05-31.23
 
 # You should have received a copy of the GNU General Public License
 # along with this program; if not, write to the Free Software
-# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
-# 02111-1307, USA.
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
+# 02110-1301, USA.
 
 # As a special exception to the GNU General Public License, if you
 # distribute this file as part of a program that contains a
@@ -50,11 +50,11 @@ Environment variables:
 
 Report bugs to <bug-automake@gnu.org>.
 EOF
-    exit 0
+    exit $?
     ;;
   -v | --v*)
     echo "depcomp $scriptversion"
-    exit 0
+    exit $?
     ;;
 esac
 
@@ -276,6 +276,29 @@ icc)
   rm -f "$tmpdepfile"
   ;;
 
+ia64hp)
+  # The "hp" stanza above does not work with HP's ia64 compilers,
+  # which have integrated preprocessors.  The correct option to use
+  # with these is +Maked; it writes dependencies to a file named
+  # 'foo.d', which lands next to the object file, wherever that
+  # happens to be.
+  tmpdepfile=`echo "$object" | sed -e 's/\.o$/.d/'`
+  "$@" +Maked
+  stat=$?
+  if test $stat -eq 0; then :
+  else
+     rm -f "$tmpdepfile"
+     exit $stat
+  fi
+  rm -f "$depfile"
+
+  # The object file name is correct already.
+  cat "$tmpdepfile" > "$depfile"
+  # Add `dependent.h:' lines.
+  sed -ne '2,${; s/^ //; s/ \\*$//; s/$/:/; p; }' "$tmpdepfile" >> "$depfile"
+  rm -f "$tmpdepfile"
+  ;;
+
 tru64)
    # The Tru64 compiler uses -MD to generate dependencies as a side
    # effect.  `cc -MD -o foo.o ...' puts the dependencies into `foo.o.d'.
@@ -287,36 +310,43 @@ tru64)
    base=`echo "$object" | sed -e 's|^.*/||' -e 's/\.o$//' -e 's/\.lo$//'`
 
    if test "$libtool" = yes; then
-      # Dependencies are output in .lo.d with libtool 1.4.
-      # With libtool 1.5 they are output both in $dir.libs/$base.o.d
-      # and in $dir.libs/$base.o.d and $dir$base.o.d.  We process the
-      # latter, because the former will be cleaned when $dir.libs is
-      # erased.
-      tmpdepfile1="$dir.libs/$base.lo.d"
-      tmpdepfile2="$dir$base.o.d"
-      tmpdepfile3="$dir.libs/$base.d"
+      # With Tru64 cc, shared objects can also be used to make a
+      # static library.  This mecanism is used in libtool 1.4 series to
+      # handle both shared and static libraries in a single compilation.
+      # With libtool 1.4, dependencies were output in $dir.libs/$base.lo.d.
+      #
+      # With libtool 1.5 this exception was removed, and libtool now
+      # generates 2 separate objects for the 2 libraries.  These two
+      # compilations output dependencies in in $dir.libs/$base.o.d and
+      # in $dir$base.o.d.  We have to check for both files, because
+      # one of the two compilations can be disabled.  We should prefer
+      # $dir$base.o.d over $dir.libs/$base.o.d because the latter is
+      # automatically cleaned when .libs/ is deleted, while ignoring
+      # the former would cause a distcleancheck panic.
+      tmpdepfile1=$dir.libs/$base.lo.d   # libtool 1.4
+      tmpdepfile2=$dir$base.o.d          # libtool 1.5
+      tmpdepfile3=$dir.libs/$base.o.d    # libtool 1.5
+      tmpdepfile4=$dir.libs/$base.d      # Compaq CCC V6.2-504
       "$@" -Wc,-MD
    else
-      tmpdepfile1="$dir$base.o.d"
-      tmpdepfile2="$dir$base.d"
-      tmpdepfile3="$dir$base.d"
+      tmpdepfile1=$dir$base.o.d
+      tmpdepfile2=$dir$base.d
+      tmpdepfile3=$dir$base.d
+      tmpdepfile4=$dir$base.d
       "$@" -MD
    fi
 
    stat=$?
    if test $stat -eq 0; then :
    else
-      rm -f "$tmpdepfile1" "$tmpdepfile2" "$tmpdepfile3"
+      rm -f "$tmpdepfile1" "$tmpdepfile2" "$tmpdepfile3" "$tmpdepfile4"
       exit $stat
    fi
 
-   if test -f "$tmpdepfile1"; then
-      tmpdepfile="$tmpdepfile1"
-   elif test -f "$tmpdepfile2"; then
-      tmpdepfile="$tmpdepfile2"
-   else
-      tmpdepfile="$tmpdepfile3"
-   fi
+   for tmpdepfile in "$tmpdepfile1" "$tmpdepfile2" "$tmpdepfile3" "$tmpdepfile4"
+   do
+     test -f "$tmpdepfile" && break
+   done
    if test -f "$tmpdepfile"; then
       sed -e "s,^.*\.[a-z]*:,$object:," < "$tmpdepfile" > "$depfile"
       # That's a tab and a space in the [].
@@ -460,7 +490,8 @@ cpp)
   done
 
   "$@" -E |
-    sed -n '/^# [0-9][0-9]* "\([^"]*\)".*/ s:: \1 \\:p' |
+    sed -n -e '/^# [0-9][0-9]* "\([^"]*\)".*/ s:: \1 \\:p' \
+       -e '/^#line [0-9][0-9]* "\([^"]*\)".*/ s:: \1 \\:p' |
     sed '$ s: \\$::' > "$tmpdepfile"
   rm -f "$depfile"
   echo "$object : \\" > "$depfile"
===================================================================
Index: config/depstand.m4
--- config/depstand.m4	1 Jan 1970 00:00:00 -0000
+++ config/depstand.m4	7 Jun 2005 18:56:53 -0000
@@ -0,0 +1,136 @@
+##                                                          -*- Autoconf -*-
+
+# Copyright (C) 1999, 2000, 2001, 2002, 2003, 2004, 2005
+# Free Software Foundation, Inc.
+#
+# This file is free software; the Free Software Foundation
+# gives unlimited permission to copy and/or distribute it,
+# with or without modifications, as long as this notice is preserved.
+
+# serial 8
+
+# Based on depend.m4 from automake 1.9, modified for standalone use in
+# an environment where GNU make is required. 
+
+# ZW_PROG_COMPILER_DEPENDENCIES
+# -----------------------------
+# Variant of _AM_DEPENDENCIES which just does the dependency probe and
+# sets fooDEPMODE accordingly.  Cache-variable compatible with
+# original; not side-effect compatible.  As the users of this macro
+# may require accurate dependencies for correct builds, it does *not*
+# honor --disable-dependency-checking, and failure to detect a usable
+# method is an error.  depcomp is assumed to be located in
+# $ac_aux_dir.
+#
+# FIXME: Should use the Autoconf 2.5x language-selection mechanism.
+
+AC_DEFUN([ZW_PROG_COMPILER_DEPENDENCIES],
+[ifelse([$1], CC,   [depcc="$CC"   am_compiler_list=],
+        [$1], CXX,  [depcc="$CXX"  am_compiler_list=],
+        [$1], OBJC, [depcc="$OBJC" am_compiler_list='gcc3 gcc'],
+        [$1], GCJ,  [depcc="$GCJ"  am_compiler_list='gcc3 gcc'],
+                    [depcc="$$1"   am_compiler_list=])
+
+am_depcomp=$ac_aux_dir/depcomp
+AC_CACHE_CHECK([dependency style of $depcc],
+               [am_cv_$1_dependencies_compiler_type],
+[if test -f "$am_depcomp"; then
+  # We make a subdir and do the tests there.  Otherwise we can end up
+  # making bogus files that we don't know about and never remove.  For
+  # instance it was reported that on HP-UX the gcc test will end up
+  # making a dummy file named `D' -- because `-MD' means `put the output
+  # in D'.
+  mkdir conftest.dir
+  # Copy depcomp to subdir because otherwise we won't find it if we're
+  # using a relative directory.
+  cp "$am_depcomp" conftest.dir
+  cd conftest.dir
+  # We will build objects and dependencies in a subdirectory because
+  # it helps to detect inapplicable dependency modes.  For instance
+  # both Tru64's cc and ICC support -MD to output dependencies as a
+  # side effect of compilation, but ICC will put the dependencies in
+  # the current directory while Tru64 will put them in the object
+  # directory.
+  mkdir sub
+
+  am_cv_$1_dependencies_compiler_type=none
+  if test "$am_compiler_list" = ""; then
+     am_compiler_list=`sed -n ['s/^\([a-zA-Z0-9]*\))$/\1/p'] < ./depcomp`
+  fi
+  for depmode in $am_compiler_list; do
+    if test $depmode = none; then break; fi
+
+    _AS_ECHO([$as_me:$LINENO: trying $depmode], AS_MESSAGE_LOG_FD)
+    # Setup a source with many dependencies, because some compilers
+    # like to wrap large dependency lists on column 80 (with \), and
+    # we should not choose a depcomp mode which is confused by this.
+    #
+    # We need to recreate these files for each test, as the compiler may
+    # overwrite some of them when testing with obscure command lines.
+    # This happens at least with the AIX C compiler.
+    : > sub/conftest.c
+    for i in 1 2 3 4 5 6; do
+      echo '#include "conftst'$i'.h"' >> sub/conftest.c
+      # Using `: > sub/conftst$i.h' creates only sub/conftst1.h with
+      # Solaris 8's {/usr,}/bin/sh.
+      touch sub/conftst$i.h
+    done
+    echo "include sub/conftest.Po" > confmf
+
+    # We check with `-c' and `-o' for the sake of the "dashmstdout"
+    # mode.  It turns out that the SunPro C++ compiler does not properly
+    # handle `-M -o', and we need to detect this.
+    depcmd="depmode=$depmode \
+       source=sub/conftest.c object=sub/conftest.${OBJEXT-o} \
+       depfile=sub/conftest.Po tmpdepfile=sub/conftest.TPo \
+       $SHELL ./depcomp $depcc -c -o sub/conftest.${OBJEXT-o} sub/conftest.c"
+    echo "| $depcmd" | sed -e 's/  */ /g' >&AS_MESSAGE_LOG_FD
+    if env $depcmd > conftest.err 2>&1 &&
+       grep sub/conftst6.h sub/conftest.Po >>conftest.err 2>&1 &&
+       grep sub/conftest.${OBJEXT-o} sub/conftest.Po >>conftest.err 2>&1 &&
+       ${MAKE-make} -s -f confmf >>conftest.err 2>&1; then
+      # icc doesn't choke on unknown options, it will just issue warnings
+      # or remarks (even with -Werror).  So we grep stderr for any message
+      # that says an option was ignored or not supported.
+      # When given -MP, icc 7.0 and 7.1 complain thusly:
+      #   icc: Command line warning: ignoring option '-M'; no argument required
+      # The diagnosis changed in icc 8.0:
+      #   icc: Command line remark: option '-MP' not supported
+      if (grep 'ignoring option' conftest.err ||
+          grep 'not supported' conftest.err) >/dev/null 2>&1; then :; else
+        am_cv_$1_dependencies_compiler_type=$depmode
+	_AS_ECHO([$as_me:$LINENO: success], AS_MESSAGE_LOG_FD)
+        break
+      fi
+    fi
+    _AS_ECHO([$as_me:$LINENO: failure, diagnostics are:], AS_MESSAGE_LOG_FD)
+    sed -e 's/^/| /' < conftest.err >&AS_MESSAGE_LOG_FD
+  done
+
+  cd ..
+  rm -rf conftest.dir
+else
+  am_cv_$1_dependencies_compiler_type=none
+fi
+])
+if test x${am_cv_$1_dependencies_compiler_type-none} = xnone
+then AC_MSG_ERROR([no usable dependency style found])
+else AC_SUBST([$1DEPMODE], [depmode=$am_cv_$1_dependencies_compiler_type])
+fi
+])
+
+# AM_SET_DEPDIR
+# -------------
+# Choose a directory name for dependency files.
+AC_DEFUN([AM_SET_DEPDIR],
+[AC_REQUIRE([AM_SET_LEADING_DOT])dnl
+AC_SUBST([DEPDIR], ["${am__leading_dot}deps"])dnl
+])
+
+# ZW_CREATE_DEPDIR
+# ----------------
+# As AM_SET_DEPDIR, but also create the directory at config.status time.
+AC_DEFUN([ZW_CREATE_DEPDIR],
+[AC_REQUIRE([AM_SET_DEPDIR])dnl
+AC_CONFIG_COMMANDS([depdir], [mkdir $DEPDIR], [DEPDIR=$DEPDIR])
+])
===================================================================
Index: config/lead-dot.m4
--- config/lead-dot.m4	1 Jan 1970 00:00:00 -0000
+++ config/lead-dot.m4	7 Jun 2005 18:56:53 -0000
@@ -0,0 +1,32 @@
+#                                                          -*- Autoconf -*-
+# Copyright (C) 2003  Free Software Foundation, Inc.
+
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2, or (at your option)
+# any later version.
+
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA
+# 02111-1307, USA.
+
+# serial 1
+
+# Check whether the underlying file-system supports filenames
+# with a leading dot.  For instance MS-DOS doesn't.
+AC_DEFUN([AM_SET_LEADING_DOT],
+[rm -rf .tst 2>/dev/null
+mkdir .tst 2>/dev/null
+if test -d .tst; then
+  am__leading_dot=.
+else
+  am__leading_dot=_
+fi
+rmdir .tst 2>/dev/null
+AC_SUBST([am__leading_dot])])
===================================================================
Index: libcpp/configure.ac
--- libcpp/configure.ac	28 May 2005 15:52:47 -0000	1.14
+++ libcpp/configure.ac	7 Jun 2005 18:57:27 -0000
@@ -29,6 +29,10 @@ ACX_PROG_CC_WARNING_ALMOST_PEDANTIC([-Wn
 # corrected.
 ACX_PROG_CC_WARNINGS_ARE_ERRORS([manual])
 
+# Dependency checking.
+ZW_CREATE_DEPDIR
+ZW_PROG_COMPILER_DEPENDENCIES([CC])
+
 # Checks for header files.
 AC_HEADER_TIME
 ACX_HEADER_STRING
===================================================================
Index: libcpp/Makefile.in
--- libcpp/Makefile.in	9 Nov 2004 21:58:43 -0000	1.13
+++ libcpp/Makefile.in	7 Jun 2005 18:57:21 -0000
@@ -51,6 +51,8 @@ RANLIB = @RANLIB@
 SHELL = @SHELL@
 USED_CATALOGS = @USED_CATALOGS@
 XGETTEXT = @XGETTEXT@
+DEPMODE = @CCDEPMODE@
+DEPDIR = @DEPDIR@
 
 datadir = @datadir@
 exec_prefix = @prefix@
@@ -60,6 +62,8 @@ prefix = @prefix@
 
 MSGMERGE = msgmerge
 mkinstalldirs = $(SHELL) $(srcdir)/../mkinstalldirs
+depcomp = $(SHELL) $(srcdir)/../depcomp
+
 INCLUDES = -I$(srcdir) -I. -I$(srcdir)/../include @INCINTL@ \
 	-I$(srcdir)/include
 
@@ -165,7 +169,8 @@ clean: mostlyclean
 distclean: clean
 	-rm -f config.h stamp-h1 config.status config.cache config.log \
 	  configure.lineno configure.status.lineno Makefile localedir.h \
-	  localedir.hs
+	  localedir.hs $(DEPDIR)/*.Po
+	-rmdir $(DEPDIR)
 
 maintainer-clean: distclean
 	@echo "This command is intended for maintainers to use"
@@ -186,10 +191,19 @@ update-po: $(CATALOGS:.gmo=.pox)
   maintainer-clean check installcheck dvi html info install-info \
   install-man update-po
 
+# Dependency rule.
+COMPILE.base = $(CC) $(DEFS) $(INCLUDES) $(CPPFLAGS) $(ALL_CFLAGS) -c
+ifeq ($(DEPMODE),depmode=gcc3)
+COMPILE = $(COMPILE.base) -o $@ -MT $@ -MD -MP -MF $(DEPDIR)/$*.Po
+else
+COMPILE = source='$<' object='$@' libtool=no DEPDIR=$(DEPDIR) $(DEPMODE) \
+	  $(depcomp) $(COMPILE.base)
+endif
+
 # Implicit rules and I18N
 
 .c.o:
-	$(CC) $(DEFS) $(INCLUDES) $(CPPFLAGS) $(ALL_CFLAGS) -c $<
+	$(COMPILE) $<
 
 # N.B. We do not attempt to copy these into $(srcdir).
 .po.gmo:
@@ -227,35 +241,7 @@ po/$(PACKAGE).pot: $(libcpp_a_SOURCES)
 .NOEXPORT:
 
 # Dependencies
+-include $(patsubst %.o, $(DEPDIR)/%.Po, $(libcpp_a_OBJS) $(makedepend_OBJS))
 
-top_srcdir = $(srcdir)/..
-top_incdir = $(srcdir)/../include
-libcpp_incdir = $(srcdir)/../libcpp/include
-
-COMMON_DEPS = config.h $(srcdir)/system.h $(srcdir)/internal.h \
-  $(top_incdir)/libiberty.h $(top_incdir)/ansidecl.h \
-  $(top_incdir)/filenames.h $(top_incdir)/safe-ctype.h \
-  $(libcpp_incdir)/symtab.h $(top_incdir)/obstack.h \
-  $(libcpp_incdir)/line-map.h $(libcpp_incdir)/cpplib.h \
-  $(libcpp_incdir)/cpp-id-data.h $(top_incdir)/getopt.h 
-
-charset.o: $(srcdir)/charset.c $(COMMON_DEPS) $(srcdir)/ucnid.h
-directives.o: $(srcdir)/directives.c $(COMMON_DEPS) $(libcpp_incdir)/mkdeps.h
-errors.o: $(srcdir)/errors.c $(COMMON_DEPS)
-expr.o: $(srcdir)/expr.c $(COMMON_DEPS)
-
-files.o: $(srcdir)/files.c $(COMMON_DEPS) $(libcpp_incdir)/mkdeps.h \
-   $(top_incdir)/hashtab.h $(top_incdir)/md5.h
-identifiers.o: $(srcdir)/identifiers.c $(COMMON_DEPS)
-init.o: $(srcdir)/init.c $(COMMON_DEPS) $(libcpp_incdir)/mkdeps.h localedir.h
-lex.o: $(srcdir)/lex.c $(COMMON_DEPS)
-line-map.o: $(srcdir)/line-map.c $(COMMON_DEPS)
-macro.o: $(srcdir)/macro.c $(COMMON_DEPS)
-makedepend.o: $(srcdir)/makedepend.c $(COMMON_DEPS) $(libcpp_incdir)/mkdeps.h
-mkdeps.o: $(srcdir)/mkdeps.c $(COMMON_DEPS)
-
-pch.o: $(srcdir)/pch.c $(COMMON_DEPS) \
-   $(top_incdir)/hashtab.h $(libcpp_incdir)/mkdeps.h
-
-symtab.o: $(srcdir)/symtab.c $(COMMON_DEPS)
-traditional.o: $(srcdir)/traditional.c $(COMMON_DEPS)
+# Dependencies on generated headers have to be explicit.
+init.o: localedir.h