Re: [distcc] gcc bootstraps with distcc

[Alexandre Oliva <oliva at lsd dot ic dot unicamp dot br>]

On Jun 19, 2003, mbp@sourcefrog.net wrote:

> On 18 Jun 2003, Dara Hazeghi <dhazeghi@yahoo.com> wrote:

>> it's good to see that distcc just keeps improving! I
>> had one question. In the known problems list, it is
>> mentioned (or rather implied) that distcc can't be
>> used for gcc bootstraps. Is this still the case? If
>> so, perhaps a bug can be opened against gcc?

> I haven't tried it.  My impression was that you could make it work by
> excluding localhost from the list, though you might also need to put
> the build directory on a shared filesystem so that the later compilers
> can be found.  I think Alexandre might know.

Here's a patch for GCC that I've just tested, that was enough to get
GCC to bootstrap using distcc, using distccrel (attached) as per the
instructions in it.

It seems like the other problem I had before, of getting different
line numbers depending on whether separate or integrated preprocessing
is done is no longer present.  Wheee!

So, arranging for the directory where preprocessing happened (before
distcc ships the preprocessed code to the remote machine which starts
gcc in a potentially-different cwd) made it to the debugging info was
the only remaining issue, and this issue is fixed with the patch
below.

The patch introduces -Mpwd, that causes GCC to output the
preprocessing directory in the preprocessed output.  This means you
have to add -Mpwd to BOOT_CFLAGS to get a successful bootstrap, as
well as to get gdb to find source files in their original location,
even if using only ccache but using several different yet similar
source and build trees.  The reasons -Mpwd is not enabled by default
are backward compatibility (the <directory> directive might be
unexpected, even though it's in a format that is entirely
backward-compatible), efficiency and to enable ccache caches to be
shared by multiple build trees.

Anyway, here's the patch and the scripts I'm using for the build.

Note that, without this patch, distccrel still works for GCC
bootstraps, as long as every distcc in the network is started from a
pwd that expands to the same directory name (even if physically
different directories), and DISTCC_LOCALHOST is set to the local
machine's name, such that distcc doesn't use the integrated
preprocessor.

Bootstrapped in mainline on athlon-pc-linux-gnu.  Ok to install?

Index: gcc/ChangeLog
from  Alexandre Oliva  <aoliva@redhat.com>

	* c.opt: Introduce -Mpwd.
	* gcc.c (cpp_unique_options): Pass it.
	* c-opts.c (c_common_handle_options): Set...
	* cpplib.h (struct cpp_options): ... current_directory option.
	* cppinit.c (read_original_filename): Return bool to tell
	whether the current directory was present in the input stream.
	Call...
	(read_original_directory): New function.  Look for a # line
	"<directory>..." right after the original filename.
	(cpp_read_main_file): Emit <directory> directive if it was not
	emitted by read_original_directory and current_directory
	option is set.
	* toplev.c (src_pwd): New static variable.
	(set_src_pwd, get_src_pwd): New functions.
	* toplev.h (get_src_pwd, set_src_pwd): Declare.
	* dbxout.c (dbxout_init): Call get_src_pwd() instead of getpwd().
	* dwarf2out.c (gen_compile_unit_die): Likewise.
	* dwarfout.c (output_compile_unit_die, dwarfout_init): Likewise.
	* c-lex.c (cb_file_change): Recognize <directory> markers.
	Rename main_input_file if it's the current input_file and an
	LC_RENAME directive comes up.
	* doc/invoke.texi, doc/cpp.texi: Add -Mpwd.
	* doc/cppopts.texi: Document it.

Index: gcc/c-lex.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/c-lex.c,v
retrieving revision 1.206
diff -u -p -r1.206 c-lex.c
--- gcc/c-lex.c 30 Jun 2003 19:36:20 -0000 1.206
+++ gcc/c-lex.c 2 Jul 2003 04:13:03 -0000
@@ -242,6 +242,14 @@ fe_file_change (const struct line_map *n
 
       (*debug_hooks->end_source_file) (to_line);
     }
+  else if (new_map->reason == LC_RENAME)
+    {
+      if (strncmp (new_map->to_file, "<directory>",
+		   strlen ("<directory>")) == 0)
+	set_src_pwd (new_map->to_file + strlen ("<directory>"));
+      if (input_filename == main_input_filename)
+	main_input_filename = new_map->to_file;
+    }
 
   update_header_times (new_map->to_file);
   in_system_header = new_map->sysp != 0;
Index: gcc/c-opts.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/c-opts.c,v
retrieving revision 1.65
diff -u -p -r1.65 c-opts.c
--- gcc/c-opts.c 30 Jun 2003 19:36:20 -0000 1.65
+++ gcc/c-opts.c 2 Jul 2003 04:13:04 -0000
@@ -350,6 +350,10 @@ c_common_handle_option (size_t scode, co
       cpp_opts->deps.phony_targets = true;
       break;
 
+    case OPT_Mpwd:
+      cpp_opts->current_directory = true;
+      break;
+
     case OPT_MQ:
     case OPT_MT:
       deps_seen = true;
Index: gcc/c.opt
===================================================================
RCS file: /cvs/gcc/gcc/gcc/c.opt,v
retrieving revision 1.5
diff -u -p -r1.5 c.opt
--- gcc/c.opt 16 Jun 2003 05:47:04 -0000 1.5
+++ gcc/c.opt 2 Jul 2003 04:13:04 -0000
@@ -107,6 +107,9 @@ C ObjC C++ ObjC++ Separate
 MP
 C ObjC C++ ObjC++
 
+Mpwd
+C ObjC C++ ObjC++
+
 MQ
 C ObjC C++ ObjC++ Joined Separate
 
Index: gcc/cppinit.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/cppinit.c,v
retrieving revision 1.284
diff -u -p -r1.284 cppinit.c
--- gcc/cppinit.c 17 Jun 2003 06:17:42 -0000 1.284
+++ gcc/cppinit.c 2 Jul 2003 04:13:05 -0000
@@ -29,7 +29,8 @@ Foundation, 59 Temple Place - Suite 330,
 
 static void init_library (void);
 static void mark_named_operators (cpp_reader *);
-static void read_original_filename (cpp_reader *);
+static bool read_original_filename (cpp_reader *);
+static bool read_original_directory (cpp_reader *);
 static void post_options (cpp_reader *);
 
 /* If we have designated initializers (GCC >2.7) these tables can be
@@ -428,6 +429,8 @@ cpp_add_dependency_target (cpp_reader *p
 const char *
 cpp_read_main_file (cpp_reader *pfile, const char *fname)
 {
+  char *pwd;
+
   sanity_checks (pfile);
 
   post_options (pfile);
@@ -457,8 +460,24 @@ cpp_read_main_file (cpp_reader *pfile, c
 
   /* For foo.i, read the original filename foo.c now, for the benefit
      of the front ends.  */
-  if (CPP_OPTION (pfile, preprocessed))
-    read_original_filename (pfile);
+  if (CPP_OPTION (pfile, preprocessed)
+      && read_original_filename (pfile))
+    pwd = NULL;
+  else if (CPP_OPTION (pfile, current_directory))
+    pwd = getpwd ();
+  else
+    pwd = NULL;
+
+  if (pwd)
+    {
+      char *dir = concat ("<directory>", pwd, NULL);
+      const char *name = pfile->map->to_file;
+      unsigned int line = pfile->line;
+      int sysp = pfile->map->sysp;
+      
+      _cpp_do_file_change (pfile, LC_RENAME, dir, 1, 0);
+      _cpp_do_file_change (pfile, LC_RENAME, name, line, sysp);
+    }
 
   return pfile->map->to_file;
 }
@@ -467,7 +486,7 @@ cpp_read_main_file (cpp_reader *pfile, c
    handle the directive so we know the original file name.  This will
    generate file_change callbacks, which the front ends must handle
    appropriately given their state of initialization.  */
-static void
+static bool
 read_original_filename (cpp_reader *pfile)
 {
   const cpp_token *token, *token1;
@@ -484,12 +503,60 @@ read_original_filename (cpp_reader *pfil
       if (token1->type == CPP_NUMBER)
 	{
 	  _cpp_handle_directive (pfile, token->flags & PREV_WHITE);
-	  return;
+	  return read_original_directory (pfile);
+	}
+    }
+
+  /* Backup as if nothing happened.  */
+  _cpp_backup_tokens (pfile, 1);
+  return false;
+}
+
+/* For preprocessed files, if the tokens following the first filename
+   line are of the form # NUM "<directory>...", handle the directive
+   so we know the original current directory.  This will generate
+   file_change callbacks, which the front ends must handle
+   appropriately given their state of initialization.  */
+static bool
+read_original_directory (cpp_reader *pfile)
+{
+  const cpp_token *token, *token1, *token2;
+
+  /* Lex ahead; if the first tokens are of the form # NUM, then
+     process the directive, otherwise back up.  */
+  token = _cpp_lex_direct (pfile);
+  if (token->type == CPP_HASH)
+    {
+      token1 = _cpp_lex_direct (pfile);
+
+      /* If it's a #line directive, handle it.  */
+      if (token1->type == CPP_NUMBER)
+	{
+	  token2 = _cpp_lex_direct (pfile);
+	  _cpp_backup_tokens (pfile, 2);
+
+	  if (token2->type == CPP_STRING
+	      && strncmp ((const char *)token2->val.str.text, "<directory>",
+			  strlen ("<directory>")) == 0)
+	    {
+	      const char *name = pfile->map->to_file;
+	      unsigned int line = pfile->line;
+	      int sysp = pfile->map->sysp;
+
+	      _cpp_handle_directive (pfile, token->flags & PREV_WHITE);
+
+	      _cpp_do_file_change (pfile, LC_RENAME, name, line, sysp);
+	      return true;
+	    }
 	}
+      else
+	_cpp_backup_tokens (pfile, 1);
     }
 
   /* Backup as if nothing happened.  */
   _cpp_backup_tokens (pfile, 1);
+
+  return false;
 }
 
 /* This is called at the end of preprocessing.  It pops the last
Index: gcc/cpplib.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/cpplib.h,v
retrieving revision 1.258
diff -u -p -r1.258 cpplib.h
--- gcc/cpplib.h 25 Jun 2003 21:01:09 -0000 1.258
+++ gcc/cpplib.h 2 Jul 2003 04:13:05 -0000
@@ -369,6 +369,11 @@ struct cpp_options
 
   /* Nonzero means __STDC__ should have the value 0 in system headers.  */
   unsigned char stdc_0_in_system_headers;
+
+  /* Nonzero means output a <directory> line marker directive right
+     after the initial file name line marker, and before a duplicate
+     initial line marker.  */
+  unsigned char current_directory;
 };
 
 /* Call backs to cpplib client.  */
Index: gcc/dbxout.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/dbxout.c,v
retrieving revision 1.148
diff -u -p -r1.148 dbxout.c
--- gcc/dbxout.c 1 Jul 2003 19:04:18 -0000 1.148
+++ gcc/dbxout.c 2 Jul 2003 04:13:08 -0000
@@ -465,7 +465,8 @@ dbxout_init (const char *input_file_name
   /* Put the current working directory in an N_SO symbol.  */
   if (use_gnu_debug_info_extensions)
     {
-      if (!cwd && (cwd = getpwd ()) && (!*cwd || cwd[strlen (cwd) - 1] != '/'))
+      if (!cwd && (cwd = get_src_pwd ())
+	  && (!*cwd || cwd[strlen (cwd) - 1] != '/'))
 	cwd = concat (cwd, FILE_NAME_JOINER, NULL);
       if (cwd)
 	{
Index: gcc/dwarf2out.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/dwarf2out.c,v
retrieving revision 1.440
diff -u -p -r1.440 dwarf2out.c
--- gcc/dwarf2out.c 1 Jul 2003 12:17:52 -0000 1.440
+++ gcc/dwarf2out.c 2 Jul 2003 04:13:15 -0000
@@ -9475,7 +9475,7 @@ add_name_attribute (dw_die_ref die, cons
 static void
 add_comp_dir_attribute (dw_die_ref die)
 {
-  const char *wd = getpwd ();
+  const char *wd = get_src_pwd ();
   if (wd != NULL)
     add_AT_string (die, DW_AT_comp_dir, wd);
 }
Index: gcc/dwarfout.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/dwarfout.c,v
retrieving revision 1.128
diff -u -p -r1.128 dwarfout.c
--- gcc/dwarfout.c 29 Jun 2003 15:19:13 -0000 1.128
+++ gcc/dwarfout.c 2 Jul 2003 04:13:19 -0000
@@ -4043,7 +4043,7 @@ output_compile_unit_die (void *arg)
     stmt_list_attribute (LINE_BEGIN_LABEL);
 
   {
-    const char *wd = getpwd ();
+    const char *wd = get_src_pwd ();
     if (wd)
       comp_dir_attribute (wd);
   }
@@ -6119,7 +6119,7 @@ dwarfout_init (const char *main_input_fi
 	  ASM_OUTPUT_PUSH_SECTION (asm_out_file, DEBUG_SFNAMES_SECTION);
 	  ASM_OUTPUT_LABEL (asm_out_file, SFNAMES_BEGIN_LABEL);
 	  {
-	    const char *pwd = getpwd ();
+	    const char *pwd = get_src_pwd ();
 	    char *dirname;
 
 	    if (!pwd)
Index: gcc/gcc.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/gcc.c,v
retrieving revision 1.383
diff -u -p -r1.383 gcc.c
--- gcc/gcc.c 29 Jun 2003 13:53:07 -0000 1.383
+++ gcc/gcc.c 2 Jul 2003 04:13:23 -0000
@@ -759,7 +759,7 @@ static const char *cpp_unique_options =
  %{!Q:-quiet} %{nostdinc*} %{C} %{CC} %{v} %{I*} %{P} %I\
  %{MD:-MD %{!o:%b.d}%{o*:%.d%*}}\
  %{MMD:-MMD %{!o:%b.d}%{o*:%.d%*}}\
- %{M} %{MM} %{MF*} %{MG} %{MP} %{MQ*} %{MT*}\
+ %{M} %{MM} %{MF*} %{MG} %{MP} %{Mpwd} %{MQ*} %{MT*}\
  %{!E:%{!M:%{!MM:%{MD|MMD:%{o*:-MQ %*}}}}}\
  %{trigraphs} %{remap} %{g3:-dD} %{H} %C %{D*&U*&A*} %{i*} %Z %i\
  %{E|M|MM:%W{o*}}";
Index: gcc/toplev.c
===================================================================
RCS file: /cvs/gcc/gcc/gcc/toplev.c,v
retrieving revision 1.795
diff -u -p -r1.795 toplev.c
--- gcc/toplev.c 1 Jul 2003 12:18:01 -0000 1.795
+++ gcc/toplev.c 2 Jul 2003 04:13:27 -0000
@@ -1561,6 +1561,41 @@ FILE *asm_out_file;
 FILE *aux_info_file;
 FILE *rtl_dump_file = NULL;
 
+/* The current working directory of a translation.  It's generally the
+   directory from which compilation was initiated, but a preprocessed
+   file may specify the original directory in which it was
+   created.  */
+
+static const char *src_pwd;
+
+/* Initialize src_pwd with the given string, and return true.  If it
+   was already initialized, return false.  As a special case, it may
+   be called with a NULL argument to test whether src_pwd has NOT been
+   initialized yet.  */
+
+bool
+set_src_pwd (const char *pwd)
+{
+  if (src_pwd)
+    return false;
+
+  src_pwd = xstrdup (pwd);
+  return true;
+}
+
+/* Return the directory from which the translation unit was initiated,
+   in case set_src_pwd() was not called before to assign it a
+   different value.  */
+
+const char *
+get_src_pwd (void)
+{
+  if (! src_pwd)
+    src_pwd = getpwd ();
+
+   return src_pwd;
+}
+
 /* Decode the string P as an integral parameter.
    If the string is indeed an integer return its numeric value else
    issue an Invalid Option error for the option PNAME and return DEFVAL.
Index: gcc/toplev.h
===================================================================
RCS file: /cvs/gcc/gcc/gcc/toplev.h,v
retrieving revision 1.102
diff -u -p -r1.102 toplev.h
--- gcc/toplev.h 28 Jun 2003 06:18:10 -0000 1.102
+++ gcc/toplev.h 2 Jul 2003 04:13:27 -0000
@@ -161,4 +161,10 @@ extern bool fast_math_flags_set_p	(void)
 extern int exact_log2_wide             (unsigned HOST_WIDE_INT);
 extern int floor_log2_wide             (unsigned HOST_WIDE_INT);
 
+/* Functions used to get and set GCC's notion of in what directory
+   compilation was started.  */
+
+extern const char *get_src_pwd	       (void);
+extern bool set_src_pwd		       (const char *);
+
 #endif /* ! GCC_TOPLEV_H */
Index: gcc/doc/cpp.texi
===================================================================
RCS file: /cvs/gcc/gcc/gcc/doc/cpp.texi,v
retrieving revision 1.60
diff -u -p -r1.60 cpp.texi
--- gcc/doc/cpp.texi 19 May 2003 22:15:57 -0000 1.60
+++ gcc/doc/cpp.texi 2 Jul 2003 04:34:54 -0000
@@ -4087,7 +4087,8 @@ without notice.
 cpp [@option{-D}@var{macro}[=@var{defn}]@dots{}] [@option{-U}@var{macro}]
     [@option{-I}@var{dir}@dots{}] [@option{-W}@var{warn}@dots{}]
     [@option{-M}|@option{-MM}] [@option{-MG}] [@option{-MF} @var{filename}]
-    [@option{-MP}] [@option{-MQ} @var{target}@dots{}] [@option{-MT} @var{target}@dots{}]
+    [@option{-MP}] [@option{-Mpwd}] [@option{-MQ} @var{target}@dots{}]
+    [@option{-MT} @var{target}@dots{}]
     [@option{-x} @var{language}] [@option{-std=}@var{standard}]
     @var{infile} @var{outfile}
 
Index: gcc/doc/cppopts.texi
===================================================================
RCS file: /cvs/gcc/gcc/gcc/doc/cppopts.texi,v
retrieving revision 1.24
diff -u -p -r1.24 cppopts.texi
--- gcc/doc/cppopts.texi 17 May 2003 20:29:33 -0000 1.24
+++ gcc/doc/cppopts.texi 2 Jul 2003 04:34:54 -0000
@@ -1,4 +1,4 @@
-@c Copyright (c) 1999, 2000, 2001, 2002
+@c Copyright (c) 1999, 2000, 2001, 2002, 2003
 @c Free Software Foundation, Inc.
 @c This is part of the CPP and GCC manuals.
 @c For copying conditions, see the file gcc.texi.
@@ -260,6 +260,15 @@ test.o: test.c test.h
 
 test.h:
 @end example
+
+@item -Mpwd
+@opindex Mpwd
+This option instructs CPP to output, right after the initial line
+marker, a second line marker indicating the directory in which
+preprocessing occurred, and a third line marker identical to the
+first.  GCC uses this second line marker, when it's present, to
+override the current working directory, that is emitted in some
+debugging information formats.
 
 @item -MT @var{target}
 @opindex MT
Index: gcc/doc/invoke.texi
===================================================================
RCS file: /cvs/gcc/gcc/gcc/doc/invoke.texi,v
retrieving revision 1.307
diff -u -p -r1.307 invoke.texi
--- gcc/doc/invoke.texi 1 Jul 2003 14:39:20 -0000 1.307
+++ gcc/doc/invoke.texi 2 Jul 2003 04:35:04 -0000
@@ -300,7 +300,7 @@ in the following sections.
 -include @var{file}  -imacros @var{file} @gol
 -iprefix @var{file}  -iwithprefix @var{dir} @gol
 -iwithprefixbefore @var{dir}  -isystem @var{dir} @gol
--M  -MM  -MF  -MG  -MP  -MQ  -MT  -nostdinc  -P  -remap @gol
+-M  -MM  -MF  -MG  -MP  -Mpwd  -MQ  -MT  -nostdinc  -P  -remap @gol
 -trigraphs  -undef  -U@var{macro}  -Wp,@var{option} @gol
 -Xpreprocessor @var{option}}

Attachment: distccrel
Description: Binary data

Attachment: netpwd
Description: Binary data

-- 
Alexandre Oliva   Enjoy Guarana', see http://www.ic.unicamp.br/~oliva/
Red Hat GCC Developer                 aoliva@{redhat.com, gcc.gnu.org}
CS PhD student at IC-Unicamp        oliva@{lsd.ic.unicamp.br, gnu.org}
Free Software Evangelist                Professional serial bug killer