[jit] Experimental in-process embedding of the gcc driver into the jit
David Malcolm
dmalcolm@redhat.com
Fri Sep 26 20:31:00 GMT 2014
On Tue, 2014-09-23 at 23:27 +0000, Joseph S. Myers wrote:
[...]
> The code for compiling a .s file should:
[...]
> * use libiberty's pexecute to run subprocesses, not "system" (building up
> a string to pass to the shell always looks like a security hole, though in
> this case it may in fact be safe);
>
> * use the $(target_noncanonical)-gcc-$(version) name for the driver rather
> than plain "gcc", to maximise the chance that it is actually the same
> compiler the JIT library was built for (I realise you may not actually
> depend on it being the same compiler, but that does seem best; in
> principle in future it should be possible to load multiple copies of the
> JIT library to JIT for different targets, so that code for an offload
> accelerator can go through the JIT).
[...]
The JIT generates assembler, but needs to generate a shared library.
Currently it invokes a "gcc" driver binary to go from .s to .so
I had the idea of turning the driver code in gcc.c into a library
and using it directly, in-process.
This experiment renames "main" in gcc.c to "driver_main" for use by
the insides of the JIT library, adding a gcc-main.c with a "main" that
simply calls "driver_main" (rather like how we have main.c calling
toplev_main for use by cc1 etc).
I can then call driver_main from inside the JIT library, and call a
new "driver_finalize" function to try to cleanup state in gcc.c enough
to support repeated calls.
I have to set LIBRARY_PATH so that the "ln" invocation can find -lgcc
and -lgcc_s:
LD_LIBRARY_PATH=. \
LIBRARY_PATH=../../install/lib/gcc/x86_64-unknown-linux-gnu/5.0.0:../../install/lib \
gdb --args \
testsuite/jit/test-factorial.exe
I also pass -fno-use-linker-plugin to driver_main to stop path issues
locating that.
This works for 5 or 6 in-process iterations, but eventually dies with:
test-factorial.exe: error trying to exec 'ld': execvp: Argument list too long
LIBRARY_PATH in the process' environment gets crazily long; what I think
is happening is that gcc.c uses getenv("LIBRARY_PATH"), processes the result
somewhat, then uses putenv("LIBRARY_PATH"), leading to (I think) an
exponential explosion in the length of LIBRARY_PATH in the process's env
(and the eventual failure seen above).
Other than that... my simple testcase seems to work.
In crude perftesting it currently seems to be slightly *slower*; e.g.:
Using driver_main, buggily:
assemble JIT code : 0.00 ( 0%) usr 0.04 (80%) sys 0.18 (49%) wall 0 kB ( 0%) ggc
TOTAL : 0.16 0.05 0.37 2348 kB
Using a subprocess:
assemble JIT code : 0.00 ( 0%) usr 0.00 ( 0%) sys 0.15 (48%) wall 0 kB ( 0%) ggc
TOTAL : 0.14 0.02 0.31 2348 kB
Perhaps this is because of the env accumulation bug above, or maybe I'm
doing an apples-to-oranges comparison somewhere.
Not committing this; just posting it for discussion and archival.
gcc/ChangeLog.jit:
* Makefile.in (GCC_OBJS): Add gcc-main.o.
* gcc/gcc-main.c: New file, implementing just a "main".
* gcc.c (main): Rename to...
(driver_main): ...this.
(driver_finalize): New function.
* gcc.h (driver_main): New prototype.
(driver_finalize): Likewise.
gcc/jit/ChangeLog.jit:
* Make-lang.in (jit_OBJS): Add gcc.o so that we can use
driver_main, and jitspec.o to provide implementations of functions
needed by it. Put .o files on individual lines, sorted
alphabetically.
(LIBGCCJIT_FILENAME): Add $(EXTRA_GCC_OBJS) $(EXTRA_GCC_LIBS) to
the linkage line (and deps), so that we can pick up the
config's implementation of "host_detect_local_cpu", which is
needed by gcc.o.
* internal-api.c: Include gcc.h.
(gcc::jit::playback::context::compile): Rewrite invocation of
assembler and linker to simply call into driver_main in-process,
rather than invoking "gcc". Keep the old code around for now
for performance testing.
* jitspec.c: New file, providing implementations of functions
and variables needed by gcc.o: functions lang_specific_driver
and lang_specific_pre_link, and variable
lang_specific_extra_outfiles.
* notes.txt: Show the invocation of driver_main and
driver_finalize.
---
gcc/Makefile.in | 2 +-
gcc/gcc-main.c | 31 +++++++++++++++++++++++++++++++
gcc/gcc.c | 20 +++++++++++++++++---
gcc/gcc.h | 7 +++++++
gcc/jit/Make-lang.in | 11 +++++++++--
gcc/jit/internal-api.c | 38 +++++++++++++++++++++++++++++++++-----
gcc/jit/jitspec.c | 42 ++++++++++++++++++++++++++++++++++++++++++
gcc/jit/notes.txt | 11 +++++++++--
8 files changed, 149 insertions(+), 13 deletions(-)
create mode 100644 gcc/gcc-main.c
create mode 100644 gcc/jit/jitspec.c
diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index f56fa96..151d967 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1140,7 +1140,7 @@ CXX_TARGET_OBJS=@cxx_target_objs@
FORTRAN_TARGET_OBJS=@fortran_target_objs@
# Object files for gcc many-languages driver.
-GCC_OBJS = gcc.o ggc-none.o
+GCC_OBJS = gcc.o gcc-main.o ggc-none.o
c-family-warn = $(STRICT_WARN)
diff --git a/gcc/gcc-main.c b/gcc/gcc-main.c
new file mode 100644
index 0000000..923d746
--- /dev/null
+++ b/gcc/gcc-main.c
@@ -0,0 +1,31 @@
+/* Compiler driver program that can handle many languages.
+ Copyright (C) 1987-2014 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3. If not see
+<http://www.gnu.org/licenses/>. */
+
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "gcc.h"
+
+extern int main (int, char **);
+
+int
+main (int argc, char **argv)
+{
+ return driver_main (argc, argv);
+}
diff --git a/gcc/gcc.c b/gcc/gcc.c
index c550d9d..0d9de35 100644
--- a/gcc/gcc.c
+++ b/gcc/gcc.c
@@ -6369,10 +6369,8 @@ compare_files (char *cmpfile[])
return ret;
}
-extern int main (int, char **);
-
int
-main (int argc, char **argv)
+driver_main (int argc, char **argv)
{
size_t i;
int value;
@@ -8776,3 +8774,19 @@ convert_white_space (char *orig)
else
return orig;
}
+
+void
+driver_finalize (void)
+{
+ params_c_finalize ();
+
+ user_specs_head = NULL;
+ user_specs_tail = NULL;
+
+ mdswitches = NULL;
+ n_mdswitches = 0;
+
+ switches = NULL;
+ n_switches = 0;
+ n_switches_alloc = 0;
+}
diff --git a/gcc/gcc.h b/gcc/gcc.h
index c4a27a8..7c4b680 100644
--- a/gcc/gcc.h
+++ b/gcc/gcc.h
@@ -55,4 +55,11 @@ extern int lang_specific_extra_outfiles;
extern const char **outfiles;
+/* Entrypoint to the driver. */
+extern int driver_main (int argc, char **argv);
+
+/* Clean up state, for those that want to call the driver multiple times
+ in one process. */
+extern void driver_finalize (void);
+
#endif /* ! GCC_GCC_H */
diff --git a/gcc/jit/Make-lang.in b/gcc/jit/Make-lang.in
index 26dd022..03b7839 100644
--- a/gcc/jit/Make-lang.in
+++ b/gcc/jit/Make-lang.in
@@ -56,8 +56,13 @@ jit: $(LIBGCCJIT_FILENAME) $(LIBGCCJIT_SYMLINK) $(LIBGCCJIT_LINKER_NAME_SYMLINK)
# Tell GNU make to ignore these if they exist.
.PHONY: jit
-jit_OBJS = attribs.o jit/dummy-frontend.o jit/libgccjit.o jit/internal-api.o \
- jit/jit-builtins.o
+jit_OBJS = attribs.o \
+ gcc.o \
+ jit/dummy-frontend.o \
+ jit/internal-api.o \
+ jit/jit-builtins.o \
+ jit/jitspec.o \
+ jit/libgccjit.o
# Use strict warnings for this front end.
jit-warn = $(STRICT_WARN)
@@ -65,12 +70,14 @@ jit-warn = $(STRICT_WARN)
# We avoid using $(BACKEND) from Makefile.in in order to avoid pulling
# in main.o
$(LIBGCCJIT_FILENAME): $(jit_OBJS) \
+ $(EXTRA_GCC_OBJS) $(EXTRA_GCC_LIBS) \
libbackend.a libcommon-target.a libcommon.a \
$(CPPLIB) $(LIBDECNUMBER) \
$(LIBDEPS) $(srcdir)/jit/libgccjit.map
+$(LLINKER) $(ALL_LINKERFLAGS) $(LDFLAGS) -o $@ -shared \
$(jit_OBJS) libbackend.a libcommon-target.a libcommon.a \
$(CPPLIB) $(LIBDECNUMBER) $(LIBS) $(BACKENDLIBS) \
+ $(EXTRA_GCC_OBJS) $(EXTRA_GCC_LIBS) \
-Wl,--version-script=$(srcdir)/jit/libgccjit.map \
-Wl,-soname,$(LIBGCCJIT_SONAME)
diff --git a/gcc/jit/internal-api.c b/gcc/jit/internal-api.c
index 8ef9af9..3fe543b 100644
--- a/gcc/jit/internal-api.c
+++ b/gcc/jit/internal-api.c
@@ -43,6 +43,7 @@ along with GCC; see the file COPYING3. If not see
#include "stor-layout.h"
#include "print-tree.h"
#include "gimplify.h"
+#include "gcc.h"
#include <pthread.h>
@@ -4995,11 +4996,37 @@ compile ()
if (get_bool_option (GCC_JIT_BOOL_OPTION_DUMP_GENERATED_CODE))
dump_generated_code ();
- /* Gross hacks follow:
- We have a .s file; we want a .so file.
- We could reuse parts of gcc/gcc.c to do this.
- For now, just use the /usr/bin/gcc on the system...
- */
+ /* We have a .s file; we want a .so file. */
+#if 1
+ /*
+ Directly use the driver code in-process to invoke the appropriate
+ assembler and linker. */
+ {
+ auto_timevar assemble_timevar (TV_ASSEMBLE);
+ const char *argv[7];
+ int result;
+
+ argv[0] = fake_args[0];
+ argv[1] = "-shared";
+ /* The input: assembler. */
+ argv[2] = m_path_s_file;
+ /* The output: shared library. */
+ argv[3] = "-o";
+ argv[4] = m_path_so_file;
+ argv[5] = "-fno-use-linker-plugin";
+ argv[6] = "-v";
+
+ result = driver_main (7, const_cast <char **> (argv));
+ driver_finalize ();
+
+ if (result)
+ {
+ add_error (NULL, "error invoking gcc driver");
+ return NULL;
+ }
+ }
+#else
+ /* Invoke another gcc. */
{
auto_timevar assemble_timevar (TV_ASSEMBLE);
const char *errmsg;
@@ -5041,6 +5068,7 @@ compile ()
return NULL;
}
}
+#endif
// TODO: split out assembles vs linker
diff --git a/gcc/jit/jitspec.c b/gcc/jit/jitspec.c
new file mode 100644
index 0000000..62f5153
--- /dev/null
+++ b/gcc/jit/jitspec.c
@@ -0,0 +1,42 @@
+/* jitspec.c -- Specific flags and argument handling for the driver
+ Copyright (C) 2014 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3. If not see
+<http://www.gnu.org/licenses/>. */
+
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "tm.h"
+#include "gcc.h"
+#include "opts.h"
+
+void
+lang_specific_driver (struct cl_decoded_option **in_decoded_options ATTRIBUTE_UNUSED,
+ unsigned int *in_decoded_options_count ATTRIBUTE_UNUSED,
+ int *in_added_libraries ATTRIBUTE_UNUSED)
+{
+ /* empty for the jit. */
+}
+
+/* Called before linking. Returns 0 on success and -1 on failure. */
+int lang_specific_pre_link (void) /* Not used for jit. */
+{
+ return 0;
+}
+
+/* Number of extra output files that lang_specific_pre_link may generate. */
+int lang_specific_extra_outfiles = 0; /* Not used for jit. */
diff --git a/gcc/jit/notes.txt b/gcc/jit/notes.txt
index 54dca8f..daf58a1 100644
--- a/gcc/jit/notes.txt
+++ b/gcc/jit/notes.txt
@@ -60,9 +60,16 @@ Client Code . Generated . libgccjit.so
. . . . (the middle─end and backend)
. . . . ↓
. . <───────────────────────────── end of toplev::main
- . . │ RELEASE MUTEX .
. . │ . .
- . . │ Convert assembler to DSO
+ . . V . .
+ . . --> Convert assembler to DSO:
+ . . driver_main ()
+ . . invocation of "as"
+ . . invocation of "ld"
+ . . driver_finalize ()
+ . . <----
+ . . │ . .
+ . . │ RELEASE MUTEX .
. . │ . .
. . │ Load DSO .
<─────────────────────────── . .
--
1.8.5.3
More information about the Gcc-patches
mailing list