Re: Slim lto objects

[Richard Guenther <richard dot guenther at gmail dot com>]

On Mon, Sep 26, 2011 at 8:42 PM, Jan Hubicka <hubicka@ucw.cz> wrote:
> Hi,
> this patch implements slim LTO. It is updated version of Andi's patch. ?It is
> done by terminating IPA optimization after analyzing and outputting LTO and in
> compile_file skipping stuff that outputs assembly.
>
> After some consideration I neded up with -ffat-lto-objects and
> -fno-fat-lto-objects. ?Original andi's patch used simply -flto-slim that is now
> equivalent of -flto -fno-fat-lto-objects.
>
> The reason is that having switch to enable/disable fat objects is that it
> allows to flip the default to slim LTO in the future, unlike -flto-slim does.
> I hope that after some time the default build environments will be sane enough
> to work with slim lto. I.e.
>
> 1) compilation go through GCC driver and w/o calling linker directly.
> ? This is needed to make LTO work at all.
> 2) we will eventually drop collect2 path that doesn't work with slim LTO
> ? in favour of linker plugins everywhere. This need someone to implement
> ? glue to darwin linker plugin API
> 3) ar/nm and friends updated to deal with GCC linker plugin by default.
> ? BFD already has search path for plugins, we need to install our plugin there
> ? and deal sanely with multiple versions of LTO.
>
> Incrementally I would like to do
> 1) Make testsuite to cover both slim and fat lto. ?I tested the patch with
> ? -fno-fat-lto-objects. Some testsuite updating is needed because
> ? a) we output some diagnostics still from expand and those don't come out
> ? ? ?for compile only tests with -fno-fat-lto-objects. We probably could move
> ? ? ?all this safely to end of early opts.
> ? b) we will need to detect linker plugin default and don't test
> ? ? ?-fno-fat-lto-objects
> 2) Andi has wrappers for ar/nm to enable plugin
> 3) For bootstrap with slim LTO we also need libtool update.
>
> Honza
>
> ? ? ? ?Andi Kleen ?<ak@linux.intel.com>
> ? ? ? ?Jan Hubicka ?<jh@suse.cz>
> ? ? ? ?* doc/invoke.texi (ffat-lto-objects): Document.
> ? ? ? ?* toplev.c (compile_file): Do not output assembly when doing slim lto;
> ? ? ? ?Output __gnu_slim_lto when doing slim lto.
> ? ? ? ?* cgraphunit.c (ipa_passes): Do only analysis when producing slim lto.
> ? ? ? ?(cgraph_optimize): Return early when doing slim lto.
> ? ? ? ?* opts.c (finish_options): Complain about lack of linker plugin
> ? ? ? ?when doing slim lto.
> ? ? ? ?* common.opt (ffat-lto-objects): New.
> Index: doc/invoke.texi
> ===================================================================
> *** doc/invoke.texi ? ? (revision 179188)
> --- doc/invoke.texi ? ? (working copy)
> *************** Objective-C and Objective-C++ Dialects}.
> *** 355,362 ****
> ?-fcx-limited-range @gol
> ?-fdata-sections -fdce -fdce -fdelayed-branch @gol
> ?-fdelete-null-pointer-checks -fdse -fdevirtualize -fdse @gol
> ! -fearly-inlining -fipa-sra -fexpensive-optimizations -ffast-math @gol
> ! -ffinite-math-only -ffloat-store -fexcess-precision=@var{style} @gol
> ?-fforward-propagate -ffp-contract=@var{style} -ffunction-sections @gol
> ?-fgcse -fgcse-after-reload -fgcse-las -fgcse-lm -fgraphite-identity @gol
> ?-fgcse-sm -fif-conversion -fif-conversion2 -findirect-inlining @gol
> --- 355,362 ----
> ?-fcx-limited-range @gol
> ?-fdata-sections -fdce -fdce -fdelayed-branch @gol
> ?-fdelete-null-pointer-checks -fdse -fdevirtualize -fdse @gol
> ! -fearly-inlining -fipa-sra -fexpensive-optimizations -ffat-lto-objects @gol
> ! -ffast-math -ffinite-math-only -ffloat-store -fexcess-precision=@var{style} @gol
> ?-fforward-propagate -ffp-contract=@var{style} -ffunction-sections @gol
> ?-fgcse -fgcse-after-reload -fgcse-las -fgcse-lm -fgraphite-identity @gol
> ?-fgcse-sm -fif-conversion -fif-conversion2 -findirect-inlining @gol
> *************** use it.
> *** 7891,7896 ****
> --- 7891,7912 ----
> ?Enabled by default when LTO support in GCC is enabled and GCC was compiled
> ?with a linker supporting plugins (GNU ld 2.21 or newer or gold).
>
> + @item -ffat-lto-objects
> + @opindex ffat-lto-objects
> + Fat LTO objects are object files that contain both the intermediate language
> + and the object code. This makes them useable for both LTO linking and normal
> + linking. This option makes effect only with @option{-flto} and is ignored
> + at linktime.
> +
> + @option{-fno-fat-lto-objects} improves compilation time over plain LTO, but
> + requires the complete toolchain to be aware of LTO. Requires a linker with
> + linker plugin support for basic functionality. ?Nm, ar and ranlib has to be
> + also linker plugin enabled to allow full featured build environment (capable of
> + building static libraries etc).
> +
> + The default is @option{-ffat-lto-objects} but this default is intended to
> + change in future releases when linker plugin enabled environments become more
> + common.

missing vertical space

Ok with that change.

Please also update changes.html.

Richard.

> ?@item -fcompare-elim
> ?@opindex fcompare-elim
> ?After register allocation and post-register allocation instruction splitting,
> Index: toplev.c
> ===================================================================
> *** toplev.c ? ?(revision 179188)
> --- toplev.c ? ?(working copy)
> *************** compile_file (void)
> *** 569,608 ****
> ? ? ? ?return;
> ? ? ?}
>
> ! ? varpool_assemble_pending_decls ();
> ! ? finish_aliases_2 ();
>
> ! ? /* Likewise for mudflap static object registrations. ?*/
> ! ? if (flag_mudflap)
> ! ? ? mudflap_finish_file ();
>
> ! ? output_shared_constant_pool ();
> ! ? output_object_blocks ();
>
> ! ? /* Write out any pending weak symbol declarations. ?*/
> ! ? weak_finish ();
>
> ! ? /* This must be at the end before unwind and debug info.
> ! ? ? ?Some target ports emit PIC setup thunks here. ?*/
> ! ? targetm.asm_out.code_end ();
>
> ! ? /* Do dbx symbols. ?*/
> ! ? timevar_push (TV_SYMOUT);
>
> ! #if defined DWARF2_DEBUGGING_INFO || defined DWARF2_UNWIND_INFO
> ! ? if (dwarf2out_do_frame ())
> ! ? ? dwarf2out_frame_finish ();
> ! #endif
>
> ! ? (*debug_hooks->finish) (main_input_filename);
> ! ? timevar_pop (TV_SYMOUT);
>
> ! ? /* Output some stuff at end of file if nec. ?*/
>
> ! ? dw2_output_indirect_constants ();
>
> ! ? /* Flush any pending external directives. ?*/
> ! ? process_pending_assemble_externals ();
>
> ? ?/* Emit LTO marker if LTO info has been previously emitted. ?This is
> ? ? ? used by collect2 to determine whether an object file contains IL.
> --- 569,613 ----
> ? ? ? ?return;
> ? ? ?}
>
> ! ? /* Compilation unit is finalized. ?When producing non-fat LTO object, we are
> ! ? ? ?basically finished. ?*/
> ! ? if (in_lto_p || !flag_lto || flag_fat_lto_objects)
> ! ? ? {
> ! ? ? ? varpool_assemble_pending_decls ();
> ! ? ? ? finish_aliases_2 ();
>
> ! ? ? ? /* Likewise for mudflap static object registrations. ?*/
> ! ? ? ? if (flag_mudflap)
> ! ? ? ? mudflap_finish_file ();
>
> ! ? ? ? output_shared_constant_pool ();
> ! ? ? ? output_object_blocks ();
>
> ! ? ? ? /* Write out any pending weak symbol declarations. ?*/
> ! ? ? ? weak_finish ();
>
> ! ? ? ? /* This must be at the end before unwind and debug info.
> ! ? ? ? ?Some target ports emit PIC setup thunks here. ?*/
> ! ? ? ? targetm.asm_out.code_end ();
>
> ! ? ? ? /* Do dbx symbols. ?*/
> ! ? ? ? timevar_push (TV_SYMOUT);
>
> ! ? ? #if defined DWARF2_DEBUGGING_INFO || defined DWARF2_UNWIND_INFO
> ! ? ? ? if (dwarf2out_do_frame ())
> ! ? ? ? dwarf2out_frame_finish ();
> ! ? ? #endif
>
> ! ? ? ? (*debug_hooks->finish) (main_input_filename);
> ! ? ? ? timevar_pop (TV_SYMOUT);
>
> ! ? ? ? /* Output some stuff at end of file if nec. ?*/
>
> ! ? ? ? dw2_output_indirect_constants ();
>
> ! ? ? ? /* Flush any pending external directives. ?*/
> ! ? ? ? process_pending_assemble_externals ();
> ! ? ?}
>
> ? ?/* Emit LTO marker if LTO info has been previously emitted. ?This is
> ? ? ? used by collect2 to determine whether an object file contains IL.
> *************** compile_file (void)
> *** 623,628 ****
> --- 628,650 ----
> ? ? ? ? ? ? ? ? ? ? ? ? (unsigned HOST_WIDE_INT) 1,
> ? ? ? ? ? ? ? ? ? ? ? ? (unsigned HOST_WIDE_INT) 1);
> ?#endif
> + ? ? ? /* Let linker plugin know that this is a slim object and must be LTOed
> + ? ? ? ? ?even when user did not ask for it. ?*/
> + ? ? ? if (!flag_fat_lto_objects)
> + ? ? ? ? {
> + #if defined ASM_OUTPUT_ALIGNED_DECL_COMMON
> + ? ? ? ? ASM_OUTPUT_ALIGNED_DECL_COMMON (asm_out_file, NULL_TREE,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? "__gnu_slim_lto",
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? (unsigned HOST_WIDE_INT) 1, 8);
> + #elif defined ASM_OUTPUT_ALIGNED_COMMON
> + ? ? ? ? ASM_OUTPUT_ALIGNED_COMMON (asm_out_file, "__gnu_slim_lto",
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?(unsigned HOST_WIDE_INT) 1, 8);
> + #else
> + ? ? ? ? ASM_OUTPUT_COMMON (asm_out_file, "__gnu_slim_lto",
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ?(unsigned HOST_WIDE_INT) 1,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ?(unsigned HOST_WIDE_INT) 1);
> + #endif
> + ? ? ? ? }
> ? ? ?}
>
> ? ?/* Attach a special .ident directive to the end of the file to identify
> Index: cgraphunit.c
> ===================================================================
> *** cgraphunit.c ? ? ? ?(revision 179188)
> --- cgraphunit.c ? ? ? ?(working copy)
> *************** ipa_passes (void)
> *** 2042,2048 ****
> ? ?if (flag_generate_lto)
> ? ? ?targetm.asm_out.lto_end ();
>
> ! ? if (!flag_ltrans)
> ? ? ?execute_ipa_pass_list (all_regular_ipa_passes);
> ? ?invoke_plugin_callbacks (PLUGIN_ALL_IPA_PASSES_END, NULL);
>
> --- 2042,2048 ----
> ? ?if (flag_generate_lto)
> ? ? ?targetm.asm_out.lto_end ();
>
> ! ? if (!flag_ltrans && (in_lto_p || !flag_lto || flag_fat_lto_objects))
> ? ? ?execute_ipa_pass_list (all_regular_ipa_passes);
> ? ?invoke_plugin_callbacks (PLUGIN_ALL_IPA_PASSES_END, NULL);
>
> *************** cgraph_optimize (void)
> *** 2080,2087 ****
> ? ?if (!seen_error ())
> ? ? ?ipa_passes ();
>
> ! ? /* Do nothing else if any IPA pass found errors. ?*/
> ! ? if (seen_error ())
> ? ? ?{
> ? ? ? ?timevar_pop (TV_CGRAPHOPT);
> ? ? ? ?return;
> --- 2080,2088 ----
> ? ?if (!seen_error ())
> ? ? ?ipa_passes ();
>
> ! ? /* Do nothing else if any IPA pass found errors or if we are just streaming LTO. ?*/
> ! ? if (seen_error ()
> ! ? ? ? || (!in_lto_p && flag_lto && !flag_fat_lto_objects))
> ? ? ?{
> ? ? ? ?timevar_pop (TV_CGRAPHOPT);
> ? ? ? ?return;
> Index: opts.c
> ===================================================================
> *** opts.c ? ? ?(revision 179188)
> --- opts.c ? ? ?(working copy)
> *************** finish_options (struct gcc_options *opts
> *** 779,785 ****
> ?#else
> ? ? ? ?error_at (loc, "LTO support has not been enabled in this configuration");
> ?#endif
> ! ? ? }
> ? ?if ((opts->x_flag_lto_partition_balanced != 0) + (opts->x_flag_lto_partition_1to1 != 0)
> ? ? ? ? + (opts->x_flag_lto_partition_none != 0) >= 1)
> ? ? ?{
> --- 779,787 ----
> ?#else
> ? ? ? ?error_at (loc, "LTO support has not been enabled in this configuration");
> ?#endif
> ! ? ? ? if (!opts->x_flag_fat_lto_objects && !HAVE_LTO_PLUGIN)
> ! ? ? ? ? error_at (loc, "-fno-fat-lto-objects are suppored only with linker plugin.");
> ! }
> ? ?if ((opts->x_flag_lto_partition_balanced != 0) + (opts->x_flag_lto_partition_1to1 != 0)
> ? ? ? ? + (opts->x_flag_lto_partition_none != 0) >= 1)
> ? ? ?{
> Index: common.opt
> ===================================================================
> *** common.opt ?(revision 179188)
> --- common.opt ?(working copy)
> *************** Enum(excess_precision) String(standard)
> *** 1087,1092 ****
> --- 1087,1096 ----
> ?ffast-math
> ?Common
>
> + ffat-lto-objects
> + Common Var(flag_fat_lto_objects) Init(1)
> + Output lto objects containing both the intermediate language and binary output.
> +
> ?ffinite-math-only
> ?Common Report Var(flag_finite_math_only) Optimization SetByCombined
> ?Assume no NaNs or infinities are generated
>