This is the mail archive of the gcc-patches@gcc.gnu.org mailing list for the GCC project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [LTO][PATCH] Compress LTO IL sections in object files


[Resent without text formatting due to auto-rejection of first version
by qmail at sourceware.org]

Thanks for the feedback.

Attached below is a revised version incorporating comments.  I've
confirmed that zlib is built for all front-ends by default, and that
it's side-stepped if --with-system-zlib on configure.  And a check of
cc1, jc1, objcc1, and pals shows the expected use of either gcc's
imported static zlib or the system's dynamic zlib, depending on
--with-system-zlib.

Closer?  Thanks,

--S

--
Google UK Limited | Registered Office: Belgrave House, 76 Buckingham
Palace Road, London SW1W 9TQ | Registered in England Number: 3977902
This patch provides an initial implementation of IL compression in LTO object
files.

In tests, default level compression shrinks an LTO object file to around 35%
to 40% of its prior size.  Writing IL sections with compression is broadly
neutral, and may offer a slight gain in compilation performance, as the time
taken to compress can be offset by a lower i/o overhead.  Results from
compiling one of the larger modules in gcc are shown below:

  $.../gcc-lto-branch_orig/build/gcc/xgcc -B... -c -fwhopr insn-attrtab.i
  real	0m16.769s user	0m16.232s sys	0m0.402s
  $ls -l insn-attrtab.o
  -rw-r----- 1 simonb ... 12477912 May  8 18:02 insn-attrtab.o

  $.../gcc-lto-branch_orig/build/gcc/xgcc -B... \
    -shared -fwhopr -o insn-attrtab.so insn-attrtab.o
  real	0m13.515s user	0m13.066s sys	0m0.361s
  $ls -l insn-attrtab.so
  -rwxr-x--- 1 simonb ... 1565047 May  8 18:02 insn-attrtab.so

  $.../gcc-lto-branch/build/gcc/xgcc -B... -c -fwhopr insn-attrtab.i
  real	0m15.710s user	0m15.321s sys	0m0.300s
  $ls -l insn-attrtab.o
  -rw-r----- 1 simonb ... 4993988 May  8 18:02 insn-attrtab.o

  $.../gcc-lto-branch/build/gcc/xgcc -B... \
    -shared -fwhopr -o insn-attrtab.so insn-attrtab.o
  real	0m13.859s user	0m13.269s sys	0m0.375s
  $ls -l insn-attrtab.so
  -rwxr-x--- 1 simonb ... 1565047 May  8 18:02 insn-attrtab.so

While this is a start, additional work remains.  At the moment, LTO object
files are compressed, but intermediate ltrans .lto.o files are not.  This is
due to some awkwardness in the way the LTO streamer connects to ELF i/o.
Once this is ironed out, it should be possible to carry compression through
to all LTO phases that involve serialized IL.  That said, .lto.o files are
often temporary and not necessarily held on disk, so this may not provide
much additional benefit.

The compression streams are somewhat simple, and buffer all input before
(un)compressing as the final step, rather than trying to (un)compress each
passing block.  This might be something to improve later.

And the append_data() lang hooks probably need further attention, in
particular to try to separate out the notion of data to append from the
additional pointer passed in as a block to free, perhaps delayed.

Tested and confirmed bootstraps okay on i386.  Also, confirmed gcc, g++, and
fortran testsuite parity when compared with a pre-patched build.


ChangeLog.lto
2009-05-12  Simon Baldwin  <simonb@google.com>

	* configure.ac: If --with-system-zlib, suppress local zlib and
	pass --with-system-zlib to subdir configure scripts.
	* configure: Regenerate.

gcc/ChangeLog.lto
2009-05-12  Simon Baldwin  <simonb@google.com>

	* lto-compress.h: New file.
	* lto-compress.c: New file.
	* Makefile.in: Added entries to build lto-compress module, added
	ZLIB to BACKENDLIBS.
	* lto-header.h (struct lto_stats_d): Add counters to monitor
	compression performance.
	* langhooks.c (lhd_append_data): Omit call to assemble_string if
	data to send is NULL.
	* lto-section-in.c: Include lto-compress.h.
	* (struct lto_buffer): New structure supporting data accumulation
	from compression.
	* (lto_append_data): New function, compression callback.
	* (struct lto_data_header): New structure added to uncompressed
	data streams to allow free to find the original compressed data.
	* (lto_get_section_data): Uncompress retrieved section data, and
	return instead the uncompressed text.
	* (lto_free_section_data): Retrieve the address and length of the
	original compressed data and free that, then free the uncompressed
	data.
	* lto-section-out.c: Include lto-compress.h.
	* (lto_append_data): New function, compression callback.
	* (compression_stream): Static pointer to current compression stream.
	* (lto_begin_section): Add call to lto_start_compression.
	* (lto_end_section): Add call to lto_end_compression.
	* (lto_write_stream): Add call to lto_compress_block, general tidy up
	of loop structure.
	* common.opt: Add new flag -flto-compression-level.
	* lto-utils.c (print_lto_report): Add report of (un-)compression
	statistics.

gcc/doc/ChangeLog.lto
2009-05-12  Simon Baldwin  <simonb@google.com>

	* invoke.texi: Documented -flto-compression-level.
	* sourcebuild.texi: Added information on new use of zlib.

gcc/java/ChangeLog.lto:
2009-05-12  Simon Baldwin  <simonb@google.com>

	* config-lang.in: Removed zlib from lang specific dirs.


Index: configure
===================================================================
--- configure	(revision 147288)
+++ configure	(working copy)
@@ -1137,7 +1137,7 @@ esac
     else
       echo "$as_me: WARNING: no configuration information is in $ac_dir" >&2
     fi
-    cd $ac_popdir
+    cd "$ac_popdir"
   done
 fi
 
@@ -2008,6 +2008,13 @@ if test x$with_gnu_as = xno ; then
   noconfigdirs="$noconfigdirs gas"
 fi
 
+use_system_zlib=no
+# Make sure we don't let ZLIB as be added if we didn't want it.
+if test x$with_system_zlib = xyes ; then
+  use_system_zlib=
+  noconfigdirs="$noconfigdirs zlib"
+fi
+
 # some tools are so dependent upon X11 that if we're not building with X,
 # it's not even worth trying to configure, much less build, that tool.
 
@@ -3535,8 +3542,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_c_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_c_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -3594,8 +3600,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_c_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_c_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -3711,8 +3716,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_c_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_c_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -3766,8 +3770,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_c_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_c_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -3812,8 +3815,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_c_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_c_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -3857,8 +3859,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_c_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_c_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -4042,8 +4043,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_cxx_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_cxx_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -4101,8 +4101,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_cxx_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_cxx_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -4173,8 +4172,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_cxx_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_cxx_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -4218,8 +4216,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_cxx_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_cxx_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -4644,8 +4641,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_c_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_c_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -4710,8 +4706,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_l
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_c_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_c_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -4753,8 +4748,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_l
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_c_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_c_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -4921,8 +4915,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_c_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_c_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -5040,8 +5033,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_c_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_c_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -5598,7 +5590,7 @@ if test x"${with_libs}" != x && test x"$
   fi
 fi
 
-# Set with_gnu_as and with_gnu_ld as appropriate.
+# Set with_gnu_as, with_gnu_ld, and with_system_zlib as appropriate.
 #
 # This is done by determining whether or not the appropriate directory
 # is available, and by checking whether or not specific configurations
@@ -5609,7 +5601,9 @@ fi
 #
 # If the default for a toolchain is to use GNU as and ld, and you don't
 # want to do that, then you should use the --without-gnu-as and
-# --without-gnu-ld options for the configure script.
+# --without-gnu-ld options for the configure script.  Similarly, if
+# the default is to use the included zlib and you don't want to do that,
+# you should use the --with-system-zlib option for the configure script.
 
 if test x${use_gnu_as} = x &&
    echo " ${configdirs} " | grep " gas " > /dev/null 2>&1 ; then
@@ -5623,6 +5617,12 @@ if test x${use_gnu_ld} = x &&
   extra_host_args="$extra_host_args --with-gnu-ld"
 fi
 
+if test x${use_system_zlib} = x &&
+   echo " ${configdirs} " | grep " zlib " > /dev/null 2>&1 ; then
+  with_system_zlib=yes
+  extra_host_args="$extra_host_args --with-system-zlib"
+fi
+
 # If using newlib, add --with-newlib to the extra_host_args so that gcc/configure
 # can detect this case.
 
@@ -12476,8 +12476,7 @@ if { (eval echo "$as_me:$LINENO: \"$ac_c
   cat conftest.err >&5
   echo "$as_me:$LINENO: \$? = $ac_status" >&5
   (exit $ac_status); } &&
-	 { ac_try='test -z "$ac_c_werror_flag"
-			 || test ! -s conftest.err'
+	 { ac_try='test -z "$ac_c_werror_flag"			 || test ! -s conftest.err'
   { (eval echo "$as_me:$LINENO: \"$ac_try\"") >&5
   (eval $ac_try) 2>&5
   ac_status=$?
@@ -13524,11 +13523,6 @@ esac
   *) ac_INSTALL=$ac_top_builddir$INSTALL ;;
   esac
 
-  if test x"$ac_file" != x-; then
-    { echo "$as_me:$LINENO: creating $ac_file" >&5
-echo "$as_me: creating $ac_file" >&6;}
-    rm -f "$ac_file"
-  fi
   # Let's still pretend it is `configure' which instantiates (i.e., don't
   # use $as_me), people would be surprised to read:
   #    /* config.h.  Generated by config.status.  */
@@ -13567,6 +13561,12 @@ echo "$as_me: error: cannot find input f
 	 fi;;
       esac
     done` || { (exit 1); exit 1; }
+
+  if test x"$ac_file" != x-; then
+    { echo "$as_me:$LINENO: creating $ac_file" >&5
+echo "$as_me: creating $ac_file" >&6;}
+    rm -f "$ac_file"
+  fi
 _ACEOF
 cat >>$CONFIG_STATUS <<_ACEOF
   sed "$ac_vpsub
Index: gcc/doc/sourcebuild.texi
===================================================================
--- gcc/doc/sourcebuild.texi	(revision 147288)
+++ gcc/doc/sourcebuild.texi	(working copy)
@@ -97,8 +97,9 @@ The C++ runtime library.
 Scripts used by the @code{gccadmin} account on @code{gcc.gnu.org}.
 
 @item zlib
-The @code{zlib} compression library, used by the Java front end and as
-part of the Java runtime library.
+The @code{zlib} compression library, used by the Java front end, as
+part of the Java runtime library, and for compressing and uncompressing
+gcc's intermediate language in LTO object files.
 @end table
 
 The build system in the top level directory, including how recursion
Index: gcc/doc/invoke.texi
===================================================================
--- gcc/doc/invoke.texi	(revision 147288)
+++ gcc/doc/invoke.texi	(working copy)
@@ -6853,6 +6853,14 @@ written.  This option is only meaningful
 
 Disabled by default.
 
+@item -flto-compression-level=@var{n}
+This option specified the level of compression used for intermediate language
+written to LTO object files, and is only useful when when processing object
+files in LTO mode (via -fwhopr or -flto).  Valid values are 0 to 9, and
+correspond to levels of compression provided by Zlib.  Values outside this
+range are clamped to either 0 or 9.  If the option is not given, LTO uses
+Zlib's default compression setting.
+
 @item -flto-report
 This option is only useful when processing object files in LTO
 mode (via -fwhopr or -flto).
Index: gcc/java/config-lang.in
===================================================================
--- gcc/java/config-lang.in	(revision 147288)
+++ gcc/java/config-lang.in	(working copy)
@@ -35,6 +35,6 @@ compilers="jc1\$(exeext) jvgenmain\$(exe
 gtfiles="\$(srcdir)/java/java-tree.h \$(srcdir)/java/jcf.h \$(srcdir)/java/parse.h \$(srcdir)/java/builtins.c \$(srcdir)/java/class.c \$(srcdir)/java/constants.c \$(srcdir)/java/decl.c \$(srcdir)/java/expr.c \$(srcdir)/java/jcf-parse.c \$(srcdir)/java/lang.c \$(srcdir)/java/mangle.c \$(srcdir)/java/resource.c"
 
 target_libs=${libgcj_saved}
-lang_dirs="zlib fastjar"
+lang_dirs="fastjar"
 #build_by_default=no
 lang_requires=c++
Index: gcc/lto-compress.c
===================================================================
--- gcc/lto-compress.c	(revision 0)
+++ gcc/lto-compress.c	(revision 0)
@@ -0,0 +1,304 @@
+/* LTO IL compression streams.
+
+   Copyright 2009 Free Software Foundation, Inc.
+   Contributed by Simon Baldwin <simonb@google.com>
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 3, or (at your option)
+any later version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT
+ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public
+License for more details.
+
+You should have received a copy of the GNU General Public License and
+a copy of the GCC Runtime Library Exception along with this program;
+see the files COPYING3 and COPYING.RUNTIME respectively.  If not, see
+<http://www.gnu.org/licenses/>.  */
+
+#include "config.h"
+#include "system.h"
+#include <zlib.h>
+#include "coretypes.h"
+#include "tree.h"
+#include "langhooks.h"
+#include "lto-header.h"
+#include "lto-section-out.h"
+#include "lto-compress.h"
+
+/* Compression stream structure, holds the flush callback and opaque token,
+   the buffered data, and a note of whether compression or uncompressing.  */
+
+struct lto_compression_stream
+{
+  void (*callback) (const char *, size_t, void *);
+  void *opaque;
+  char *buffer;
+  size_t bytes;
+  size_t allocation;
+  bool is_compression;
+};
+
+/* Overall compression constants for zlib.  */
+
+static const size_t Z_BUFFER_LENGTH = 4096;
+static const size_t MIN_STREAM_ALLOCATION = 1024;
+
+/* For zlib, allocate SIZE count of ITEMS and return the address, OPAQUE
+   is unused.  */
+
+static void *
+lto_zalloc (void *opaque, size_t items, size_t size)
+{
+  gcc_assert (opaque == Z_NULL);
+  return xmalloc (items * size);
+}
+
+/* For zlib, free memory at ADDRESS, OPAQUE is unused.  */
+
+static void
+lto_zfree (void *opaque, void *address)
+{
+  gcc_assert (opaque == Z_NULL);
+  free (address);
+}
+
+/* Return a zlib compression level that zlib will not reject.  Normalizes
+   the compression level from the command line flag, clamping non-default
+   values to the appropriate end of their valid range.  */
+
+static int
+lto_normalized_zlib_level (void)
+{
+  int level = flag_lto_compression_level;
+
+  if (level != Z_DEFAULT_COMPRESSION)
+    {
+      if (level < Z_NO_COMPRESSION)
+	level = Z_NO_COMPRESSION;
+      else if (level > Z_BEST_COMPRESSION)
+	level = Z_BEST_COMPRESSION;
+    }
+
+  return level;
+}
+
+/* Create a new compression stream, with CALLBACK flush function passed
+   OPAQUE token, IS_COMPRESSION indicates if compressing or uncompressing.  */
+
+static struct lto_compression_stream *
+lto_new_compression_stream (void (*callback) (const char *, size_t, void *),
+			    void *opaque, bool is_compression)
+{
+  struct lto_compression_stream *stream
+    = (struct lto_compression_stream *) xmalloc (sizeof (*stream));
+
+  memset (stream, 0, sizeof (*stream));
+  stream->callback = callback;
+  stream->opaque = opaque;
+  stream->is_compression = is_compression;
+
+  return stream;
+}
+
+/* Append NUM_CHARS from address BASE to STREAM.  */
+
+static void
+lto_append_to_compression_stream (struct lto_compression_stream *stream,
+				  const char *base, size_t num_chars)
+{
+  size_t required = stream->bytes + num_chars;
+
+  if (stream->allocation < required)
+    {
+      if (stream->allocation == 0)
+	stream->allocation = MIN_STREAM_ALLOCATION;
+      while (stream->allocation < required)
+	stream->allocation *= 2;
+
+      stream->buffer = (char *) xrealloc (stream->buffer, stream->allocation);
+    }
+
+  memcpy (stream->buffer + stream->bytes, base, num_chars);
+  stream->bytes += num_chars;
+}
+
+/* Free the buffer and memory associated with STREAM.  */
+
+static void
+lto_destroy_compression_stream (struct lto_compression_stream *stream)
+{
+  free (stream->buffer);
+  free (stream);
+}
+
+/* Return a new compression stream, with CALLBACK flush function passed
+   OPAQUE token.  */
+
+struct lto_compression_stream *
+lto_start_compression (void (*callback) (const char *, size_t, void *),
+		       void *opaque)
+{
+  return lto_new_compression_stream (callback, opaque, true);
+}
+
+/* Append NUM_CHARS from address BASE to STREAM.  */
+
+void
+lto_compress_block (struct lto_compression_stream *stream,
+		    const char *base, size_t num_chars)
+{
+  gcc_assert (stream->is_compression);
+
+  lto_append_to_compression_stream (stream, base, num_chars);
+  lto_stats.num_output_il_bytes += num_chars;
+}
+
+/* Finalize STREAM compression, and free stream allocations.  */
+
+void
+lto_end_compression (struct lto_compression_stream *stream)
+{
+  unsigned char *cursor = (unsigned char *) stream->buffer;
+  size_t remaining = stream->bytes;
+  const size_t outbuf_length = Z_BUFFER_LENGTH;
+  unsigned char *outbuf = (unsigned char *) xmalloc (outbuf_length);
+  z_stream out_stream;
+  size_t compressed_bytes = 0;
+  int status;
+
+  gcc_assert (stream->is_compression);
+
+  out_stream.next_out = outbuf;
+  out_stream.avail_out = outbuf_length;
+  out_stream.next_in = cursor;
+  out_stream.avail_in = remaining;
+  out_stream.zalloc = lto_zalloc;
+  out_stream.zfree = lto_zfree;
+  out_stream.opaque = Z_NULL;
+
+  status = deflateInit (&out_stream, lto_normalized_zlib_level ());
+  gcc_assert (status == Z_OK);
+
+  do
+    {
+      size_t in_bytes, out_bytes;
+
+      status = deflate (&out_stream, Z_FINISH);
+      gcc_assert (status == Z_OK || status == Z_STREAM_END);
+
+      in_bytes = remaining - out_stream.avail_in;
+      out_bytes = outbuf_length - out_stream.avail_out;
+
+      stream->callback ((const char *) outbuf, out_bytes, stream->opaque);
+      lto_stats.num_compressed_il_bytes += out_bytes;
+      compressed_bytes += out_bytes;
+
+      cursor += in_bytes;
+      remaining -= in_bytes;
+
+      out_stream.next_out = outbuf;
+      out_stream.avail_out = outbuf_length;
+      out_stream.next_in = cursor;
+      out_stream.avail_in = remaining;
+    }
+  while (status != Z_STREAM_END);
+
+  status = deflateEnd (&out_stream);
+  gcc_assert (status == Z_OK);
+
+  lto_destroy_compression_stream (stream);
+  free (outbuf);
+}
+
+/* Return a new uncompression stream, with CALLBACK flush function passed
+   OPAQUE token.  */
+
+struct lto_compression_stream *
+lto_start_uncompression (void (*callback) (const char *, size_t, void *),
+			 void *opaque)
+{
+  return lto_new_compression_stream (callback, opaque, false);
+}
+
+/* Append NUM_CHARS from address BASE to STREAM.  */
+
+void
+lto_uncompress_block (struct lto_compression_stream *stream,
+		      const char *base, size_t num_chars)
+{
+  gcc_assert (!stream->is_compression);
+
+  lto_append_to_compression_stream (stream, base, num_chars);
+  lto_stats.num_input_il_bytes += num_chars;
+}
+
+/* Finalize STREAM uncompression, and free stream allocations.
+  
+   Because of the way LTO IL streams are compressed, there may be several
+   concatenated compressed segments in the accumulated data, so for this
+   function we iterate decompressions until no data remains.  */
+
+void
+lto_end_uncompression (struct lto_compression_stream *stream)
+{
+  unsigned char *cursor = (unsigned char *) stream->buffer;
+  size_t remaining = stream->bytes;
+  const size_t outbuf_length = Z_BUFFER_LENGTH;
+  unsigned char *outbuf = (unsigned char *) xmalloc (outbuf_length);
+  size_t uncompressed_bytes = 0;
+
+  gcc_assert (!stream->is_compression);
+
+  while (remaining > 0)
+    {
+      z_stream in_stream;
+      size_t out_bytes;
+      int status;
+
+      in_stream.next_out = outbuf;
+      in_stream.avail_out = outbuf_length;
+      in_stream.next_in = cursor;
+      in_stream.avail_in = remaining;
+      in_stream.zalloc = lto_zalloc;
+      in_stream.zfree = lto_zfree;
+      in_stream.opaque = Z_NULL;
+
+      status = inflateInit (&in_stream);
+      gcc_assert (status == Z_OK);
+
+      do
+	{
+	  size_t in_bytes;
+
+	  status = inflate (&in_stream, Z_SYNC_FLUSH);
+	  gcc_assert (status == Z_OK || status == Z_STREAM_END);
+
+	  in_bytes = remaining - in_stream.avail_in;
+	  out_bytes = outbuf_length - in_stream.avail_out;
+
+	  stream->callback ((const char *) outbuf, out_bytes, stream->opaque);
+	  lto_stats.num_uncompressed_il_bytes += out_bytes;
+	  uncompressed_bytes += out_bytes;
+
+	  cursor += in_bytes;
+	  remaining -= in_bytes;
+
+	  in_stream.next_out = outbuf;
+	  in_stream.avail_out = outbuf_length;
+	  in_stream.next_in = cursor;
+	  in_stream.avail_in = remaining;
+	}
+      while (!(status == Z_STREAM_END && out_bytes == 0));
+
+      status = inflateEnd (&in_stream);
+      gcc_assert (status == Z_OK);
+    }
+
+  lto_destroy_compression_stream (stream);
+  free (outbuf);
+}
Index: gcc/lto-compress.h
===================================================================
--- gcc/lto-compress.h	(revision 0)
+++ gcc/lto-compress.h	(revision 0)
@@ -0,0 +1,37 @@
+/* LTO IL compression streams.
+
+   Copyright 2009 Free Software Foundation, Inc.
+   Contributed by Simon Baldwin <simonb@google.com>
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it
+under the terms of the GNU General Public License as published by
+the Free Software Foundation; either version 3, or (at your option)
+any later version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT
+ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
+or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public
+License for more details.
+
+You should have received a copy of the GNU General Public License and
+a copy of the GCC Runtime Library Exception along with this program;
+see the files COPYING3 and COPYING.RUNTIME respectively.  If not, see
+<http://www.gnu.org/licenses/>.  */
+
+struct lto_compression_stream;
+
+extern struct lto_compression_stream
+  *lto_start_compression (void (*callback) (const char *, size_t, void *),
+			  void *opaque);
+extern void lto_compress_block (struct lto_compression_stream *stream,
+				const char *base, size_t num_chars);
+extern void lto_end_compression (struct lto_compression_stream *stream);
+
+extern struct lto_compression_stream
+  *lto_start_uncompression (void (*callback) (const char *, size_t, void *),
+			    void *opaque);
+extern void lto_uncompress_block (struct lto_compression_stream *stream,
+				  const char *base, size_t num_chars);
+extern void lto_end_uncompression (struct lto_compression_stream *stream);
Index: gcc/lto-header.h
===================================================================
--- gcc/lto-header.h	(revision 147288)
+++ gcc/lto-header.h	(working copy)
@@ -95,6 +95,10 @@ struct lto_stats_d
   unsigned HOST_WIDE_INT section_size[LTO_N_SECTION_TYPES];
   unsigned HOST_WIDE_INT num_function_bodies;
   unsigned HOST_WIDE_INT num_trees[NUM_TREE_CODES];
+  unsigned HOST_WIDE_INT num_output_il_bytes;
+  unsigned HOST_WIDE_INT num_compressed_il_bytes;
+  unsigned HOST_WIDE_INT num_input_il_bytes;
+  unsigned HOST_WIDE_INT num_uncompressed_il_bytes;
 };
 
 extern struct lto_stats_d lto_stats;
Index: gcc/langhooks.c
===================================================================
--- gcc/langhooks.c	(revision 147288)
+++ gcc/langhooks.c	(working copy)
@@ -618,7 +618,8 @@ lhd_begin_section (const char *name)
 void
 lhd_append_data (const void *data, size_t len, void *block)
 {
-  assemble_string ((const char *)data, len);
+  if (data)
+    assemble_string ((const char *)data, len);
   free (block);
 }
 
Index: gcc/lto-section-in.c
===================================================================
--- gcc/lto-section-in.c	(revision 147288)
+++ gcc/lto-section-in.c	(working copy)
@@ -49,6 +49,7 @@ Boston, MA 02110-1301, USA.  */
 #include "lto-section.h"
 #include "lto-section-in.h"
 #include "lto-utils.h"
+#include "lto-compress.h"
 #include "cpplib.h"
 
 /* Section names.  These must correspond to the values of
@@ -258,6 +259,37 @@ lto_get_file_decl_data (void)
   return file_decl_data;
 }
 
+/* Buffer structure for accumulating data from compression callbacks.  */
+
+struct lto_buffer
+{
+  char *data;
+  size_t length;
+};
+
+/* Compression callback, append LENGTH bytes from DATA to the buffer pointed
+   to by OPAQUE.  */
+
+static void
+lto_append_data (const char *data, size_t length, void *opaque)
+{
+  struct lto_buffer *buffer = (struct lto_buffer *) opaque;
+
+  buffer->data = (char *) xrealloc (buffer->data, buffer->length + length);
+  memcpy (buffer->data + buffer->length, data, length);
+  buffer->length += length;
+}
+
+/* Header placed in returned uncompressed data streams.  Allows the
+   uncompressed allocated data to be mapped back to the underlying
+   compressed data for use with free_section_f.  */
+
+struct lto_data_header
+{
+  const char *data;
+  size_t len;
+};
+
 /* Return a char pointer to the start of a data stream for an LTO pass
    or function.  FILE_DATA indicates where to obtain the data.
    SECTION_TYPE is the type of information to be obtained.  NAME is
@@ -271,15 +303,44 @@ lto_get_section_data (struct lto_file_de
 		      const char *name, 
 		      size_t *len)
 {
-  const char *s = (get_section_f) (file_data, section_type, name, len);
+  const char *data = (get_section_f) (file_data, section_type, name, len);
+  const size_t header_length = sizeof (struct lto_data_header);
+  struct lto_data_header *header;
+  struct lto_buffer buffer;
+  struct lto_compression_stream *stream;
   lto_stats.section_size[section_type] += *len;
-  return s;
+
+  if (data == NULL)
+    return NULL;
+
+  /* FIXME lto: WPA mode does not write compressed sections, so for now
+     suppress uncompression if flag_ltrans.  */
+  if (flag_ltrans)
+    return data;
+
+  /* Create a mapping header containing the underlying data and length,
+     and prepend this to the uncompression buffer.  The uncompressed data
+     then follows, and a pointer to the start of the uncompressed data is
+     returned.  */
+  header = (struct lto_data_header *) xmalloc (header_length);
+  header->data = data;
+  header->len = *len;
+  
+  buffer.data = (char *) header;
+  buffer.length = header_length; 
+
+  stream = lto_start_uncompression (lto_append_data, &buffer);
+  lto_uncompress_block (stream, data, *len);
+  lto_end_uncompression (stream);
+
+  *len = buffer.length - header_length;
+  return buffer.data + header_length;
 }
 
 
-/* Return the data found from the above call.  The first three
+/* Free the data found from the above call.  The first three
    parameters are the same as above.  DATA is the data to be freed and
-   LEN is the length of that data. */
+   LEN is the length of that data.  */
 
 void 
 lto_free_section_data (struct lto_file_decl_data *file_data, 
@@ -288,8 +349,25 @@ lto_free_section_data (struct lto_file_d
 		       const char *data,
 		       size_t len)
 {
+  const size_t header_length = sizeof (struct lto_data_header);
+  const char *real_data = data - header_length;
+  const struct lto_data_header *header
+    = (const struct lto_data_header *) real_data;
+
   gcc_assert (free_section_f);
-  (free_section_f) (file_data, section_type, name, data, len);
+
+  /* FIXME lto: WPA mode does not write compressed sections, so for now
+     suppress uncompression mapping if flag_ltrans.  */
+  if (flag_ltrans)
+    {
+      (free_section_f) (file_data, section_type, name, data, len);
+      return;
+    }
+
+  /* The underlying data address has been extracted from the mapping header.
+     Free that, then free the allocated uncompression buffer.  */
+  (free_section_f) (file_data, section_type, name, header->data, header->len);
+  free (CONST_CAST (char *, real_data));
 }
 
 
Index: gcc/common.opt
===================================================================
--- gcc/common.opt	(revision 147288)
+++ gcc/common.opt	(working copy)
@@ -726,6 +726,11 @@ floop-optimize
 Common
 Does nothing.  Preserved for backward compatibility.
 
+; The initial value of -1 comes from Z_DEFAULT_COMPRESSION in zlib.h.
+flto-compression-level=
+Common Joined UInteger Var(flag_lto_compression_level) Init(-1)
+-flto-compression-level=<number> Use zlib compression level <number> for IL
+
 flto-report
 Common Report Var(flag_lto_report) Init(0) Optimization
 Report various link-time optimization statistics
Index: gcc/Makefile.in
===================================================================
--- gcc/Makefile.in	(revision 147288)
+++ gcc/Makefile.in	(working copy)
@@ -926,7 +926,7 @@ BUILD_LIBDEPS= $(BUILD_LIBIBERTY)
 # How to link with both our special library facilities
 # and the system's installed libraries.
 LIBS = @LIBS@ $(CPPLIB) $(LIBINTL) $(LIBICONV) $(LIBIBERTY) $(LIBDECNUMBER)
-BACKENDLIBS = $(CLOOGLIBS) $(PPLLIBS) $(GMPLIBS) $(PLUGINLIBS)
+BACKENDLIBS = $(CLOOGLIBS) $(PPLLIBS) $(GMPLIBS) $(PLUGINLIBS) $(ZLIB)
 # Any system libraries needed just for GNAT.
 SYSLIBS = @GNAT_LIBEXC@
 
@@ -1167,6 +1167,7 @@ OBJS-common = \
 	lto-opts.o \
 	lto-utils.o \
 	lto-wpa-fixup.o \
+	lto-compress.o \
 	mcf.o \
 	mode-switching.o \
 	modulo-sched.o \
@@ -2109,13 +2110,15 @@ lto-section-in.o: lto-section-in.c $(CON
    $(HASHTAB_H) langhooks.h $(BASIC_BLOCK_H) tree-iterator.h tree-pass.h \
    tree-flow.h $(CGRAPH_H) $(FUNCTION_H) $(GGC_H) $(DIAGNOSTIC_H) except.h \
    debug.h $(TIMEVAR_H) $(LTO_SECTION_IN_H) output.h dwarf2asm.h dwarf2out.h \
-   lto-utils.h
+   lto-utils.h lto-compress.h
 lto-section-out.o : lto-section-out.c $(CONFIG_H) $(SYSTEM_H) coretypes.h \
    $(TM_H) $(TOPLEV_H) $(TREE_H) $(EXPR_H) $(FLAGS_H) $(PARAMS_H) input.h \
    $(VARRAY_H) $(HASHTAB_H) langhooks.h $(BASIC_BLOCK_H) tree-iterator.h \
    tree-pass.h tree-flow.h $(CGRAPH_H) $(FUNCTION_H) $(GGC_H) $(DIAGNOSTIC_H) \
    except.h debug.h $(TIMEVAR_H) lto-header.h $(LTO_SECTION_OUT_H) output.h \
-   dwarf2asm.h dwarf2out.h  $(BITMAP_H) lto-utils.h $(LTO_OPTS_H)
+   dwarf2asm.h dwarf2out.h  $(BITMAP_H) lto-utils.h $(LTO_OPTS_H) lto-compress.h
+lto-compress.o: lto-compress.c $(CONFIG_H) $(SYSTEM_H) coretypes.h \
+   $(TREE_H) langhooks.h lto-header.h $(LTO_SECTION_OUT_H) lto-compress.h
 lto-symtab.o: lto-symtab.c $(LTO_TREE_IN_H) $(CONFIG_H) coretypes.h \
    $(SYSTEM_H) toplev.h $(LTO_TREE_H) $(GGC_H) $(LAMBDA_H) \
    $(GIMPLE_H) $(HASHTAB_H) $(LTO_TREE_IN_H) gt-lto-symtab.h
Index: gcc/lto-utils.c
===================================================================
--- gcc/lto-utils.c	(revision 147288)
+++ gcc/lto-utils.c	(working copy)
@@ -163,6 +163,22 @@ print_lto_report (void)
 	       HOST_WIDE_INT_PRINT_UNSIGNED "\n", s,
 	       tree_code_name[i], lto_stats.num_trees[i]);
 
+  if (flag_lto)
+    {
+      fprintf (stderr, "[%s] Compression: "
+	       HOST_WIDE_INT_PRINT_UNSIGNED " output bytes, "
+	       HOST_WIDE_INT_PRINT_UNSIGNED " compressed bytes", s,
+	       lto_stats.num_output_il_bytes,
+	       lto_stats.num_compressed_il_bytes);
+      if (lto_stats.num_output_il_bytes > 0)
+	{
+	  const float dividend = (float) lto_stats.num_compressed_il_bytes;
+	  const float divisor = (float) lto_stats.num_output_il_bytes;
+	  fprintf (stderr, " (ratio: %f)", dividend / divisor);
+	}
+      fprintf (stderr, "\n");
+    }
+
   if (flag_wpa)
     {
       fprintf (stderr, "[%s] # of output files: "
@@ -176,6 +192,19 @@ print_lto_report (void)
       fprintf (stderr, "[%s] # callgraph partitions: "
 	       HOST_WIDE_INT_PRINT_UNSIGNED "\n", s,
 	       lto_stats.num_cgraph_partitions);
+
+      fprintf (stderr, "[%s] Compression: "
+	       HOST_WIDE_INT_PRINT_UNSIGNED " input bytes, "
+	       HOST_WIDE_INT_PRINT_UNSIGNED " uncompressed bytes", s,
+	       lto_stats.num_input_il_bytes,
+	       lto_stats.num_uncompressed_il_bytes);
+      if (lto_stats.num_input_il_bytes > 0)
+	{
+	  const float dividend = (float) lto_stats.num_uncompressed_il_bytes;
+	  const float divisor = (float) lto_stats.num_input_il_bytes;
+	  fprintf (stderr, " (ratio: %f)", dividend / divisor);
+	}
+      fprintf (stderr, "\n");
     }
 
   for (i = 0; i < LTO_N_SECTION_TYPES; i++)
Index: gcc/lto-section-out.c
===================================================================
--- gcc/lto-section-out.c	(revision 147288)
+++ gcc/lto-section-out.c	(working copy)
@@ -47,6 +47,7 @@ along with GCC; see the file COPYING3.  
 #include "lto-section.h"
 #include "lto-section-out.h"
 #include "lto-tree-out.h"
+#include "lto-compress.h"
 #include "pointer-set.h"
 #include "stdint.h"
 #include "lto-symtab.h"
@@ -207,12 +208,33 @@ lto_eq_global_slot_node (const void *p1,
 *****************************************************************************/
 
 
+/* Flush compressed stream data function, sends NUM_CHARS from CHARS
+   to the append lang hook, OPAQUE is currently always NULL.  */
+
+static void
+lto_append_data (const char *chars, unsigned int num_chars, void *opaque)
+{
+  gcc_assert (opaque == NULL);
+  lang_hooks.lto.append_data (chars, num_chars, opaque);
+}
+
+/* Pointer to the current compression stream.  */
+
+static struct lto_compression_stream *compression_stream = NULL;
+
 /* Begin a new output section named NAME.  */
 
 void
 lto_begin_section (const char *name)
 {
   lang_hooks.lto.begin_section (name);
+
+  /* FIXME lto: for now, suppress compression if the lang_hook that appends
+     data is anything other than assembler output.  The effect here is that
+     we get compression of IL only in non-ltrans object files.  */
+  gcc_assert (compression_stream == NULL);
+  if (!flag_wpa)
+    compression_stream = lto_start_compression (lto_append_data, NULL);
 }
 
 
@@ -221,6 +243,11 @@ lto_begin_section (const char *name)
 void
 lto_end_section (void)
 {
+  if (compression_stream)
+    {
+      lto_end_compression (compression_stream);
+      compression_stream = NULL;
+    }
   lang_hooks.lto.end_section ();
 }
 
@@ -232,25 +259,34 @@ void
 lto_write_stream (struct lto_output_stream *obs)
 {
   unsigned int block_size = 1024;
-  unsigned int num_chars;
   struct lto_char_ptr_base *block;
+  struct lto_char_ptr_base *next_block;
   if (!obs->first_block)
     return;
 
-  block = obs->first_block;
-  while (block)
+  for (block = obs->first_block; block; block = next_block)
     {
       const char *base = ((char *)block) + sizeof (struct lto_char_ptr_base);
-      struct lto_char_ptr_base *old_block = block;
-      block = (struct lto_char_ptr_base *)block->ptr;
-      /* If there is a next block, then this one is full, if there is
-	 not a next block, then the left_in_block field says how many
-	 chars there are in this block.  */
-      num_chars = block_size - sizeof (struct lto_char_ptr_base);
-      if (!block)
-	num_chars = num_chars - obs->left_in_block;
+      unsigned int num_chars = block_size - sizeof (struct lto_char_ptr_base);
 
-      lang_hooks.lto.append_data (base, num_chars, old_block);
+      /* If this is not the last block, it is full.  If it is the last
+	 block, left_in_block indicates how many chars are unoccupied in
+	 this block; subtract from num_chars to obtain occupancy.  */
+      next_block = (struct lto_char_ptr_base *)block->ptr;
+      if (!next_block)
+	num_chars -= obs->left_in_block;
+
+      /* FIXME lto: WPA mode uses an ELF function as a lang_hook to append
+         output data.  This hook is not happy with the way that compression
+         blocks up output differently to the way it's blocked here.  So for
+         now, we don't compress WPA output.  */
+      if (compression_stream)
+	{
+	  lto_compress_block (compression_stream, base, num_chars);
+	  lang_hooks.lto.append_data (NULL, 0, block);
+	}
+      else
+	lang_hooks.lto.append_data (base, num_chars, block);
       block_size *= 2;
     }
 }
Index: configure.ac
===================================================================
--- configure.ac	(revision 147288)
+++ configure.ac	(working copy)
@@ -252,6 +252,13 @@ if test x$with_gnu_as = xno ; then
   noconfigdirs="$noconfigdirs gas"
 fi
 
+use_included_zlib=
+# Make sure we don't let ZLIB as be added if we didn't want it.
+if test x$with_system_zlib = xyes ; then
+  use_included_zlib=no
+  noconfigdirs="$noconfigdirs zlib"
+fi
+
 # some tools are so dependent upon X11 that if we're not building with X, 
 # it's not even worth trying to configure, much less build, that tool.
 
@@ -1908,7 +1915,7 @@ if test x"${with_libs}" != x && test x"$
   fi
 fi
 
-# Set with_gnu_as and with_gnu_ld as appropriate.
+# Set with_gnu_as, with_gnu_ld, and with_system_zlib as appropriate.
 #
 # This is done by determining whether or not the appropriate directory
 # is available, and by checking whether or not specific configurations
@@ -1919,7 +1926,9 @@ fi
 #
 # If the default for a toolchain is to use GNU as and ld, and you don't 
 # want to do that, then you should use the --without-gnu-as and
-# --without-gnu-ld options for the configure script.
+# --without-gnu-ld options for the configure script.  Similarly, if
+# the default is to use the included zlib and you don't want to do that,
+# you should use the --with-system-zlib option for the configure script.
 
 if test x${use_gnu_as} = x &&
    echo " ${configdirs} " | grep " gas " > /dev/null 2>&1 ; then
@@ -1933,6 +1942,12 @@ if test x${use_gnu_ld} = x &&
   extra_host_args="$extra_host_args --with-gnu-ld"
 fi
 
+if test x${use_included_zlib} = x &&
+   echo " ${configdirs} " | grep " zlib " > /dev/null 2>&1 ; then
+  with_system_zlib=yes
+  extra_host_args="$extra_host_args --with-system-zlib"
+fi
+
 # If using newlib, add --with-newlib to the extra_host_args so that gcc/configure
 # can detect this case.
 

Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]