Re: LTO/WHOPR streaming of varpool

["H.J. Lu" <hjl dot tools at gmail dot com>]

On Wed, Apr 28, 2010 at 11:09 AM, Jan Hubicka <hubicka@ucw.cz> wrote:
> Hi,
> this patch implements streaming of varpool and fixes several correctness issues
> with whopr (and makes LTO correct wrt unused variable removing too).
> Main issue with whopr was that it shipped variable declarations into multiple units
> and turned them into static vars creating large binarries (and invalid programs
> too). ?This is now fixed and WHOPR produce pretty much same size of GCC binary
> as LTO.
>
> Most of stuff in unexciting clonning of what we do for cgraph nodes already.
> I removed original logic from lto.c to figure out if variable can be accessed
> by other partition since it can not work well with WPA anyway (function bodies
> are not around and thus we don't know if inlining function will carry reference
> to variable to other unit). This is also what caused link error on several SPEC
> programs and consequently lead me to temporarily disable unused variable removal
> that result in current explossion of whopr binary size (since all variables
> are copied to every unit).
>
> There is room for cleanups left, especially in WPA, but I will do it incrementally.
> Also we should start streaming variable initializers in separate sections as we do
> for function bodies as well as list of references for each cgraph node/varpool
> to drive unused variable/function removal.
>
> Bootstrapped/regtested x86_64-linux, OK?
>
> ? ? ? ?* lto-symtab.c (lto_symtab_entry_def) Add vnode.
> ? ? ? ?(lto_varpool_replace_node): New.
> ? ? ? ?(lto_symtab_resolve_symbols): Resolve varpool nodes.
> ? ? ? ?(lto_symtab_merge_decls_1): Prefer decls with varpool node.
> ? ? ? ?(lto_symtab_merge_cgraph_nodes_1): Merge varpools.
> ? ? ? ?* cgraph.h (varpool_node_ptr): New type.
> ? ? ? ?(varpool_node_ptr): New vector.
> ? ? ? ?(varpool_node_set_def): New structure.
> ? ? ? ?(varpool_node_set): New type.
> ? ? ? ?(varpool_node_set): New vector.
> ? ? ? ?(varpool_node_set_element_def): New structure.
> ? ? ? ?(varpool_node_set_element, const_varpool_node_set_element): New types.
> ? ? ? ?(varpool_node_set_iterator): New type.
> ? ? ? ?(varpool_node): Add prev pointers, add used_from_other_partition,
> ? ? ? ?in_other_partition.
> ? ? ? ?(varpool_node_set_new, varpool_node_set_find, varpool_node_set_add,
> ? ? ? ?varpool_node_set_remove, dump_varpool_node_set, debug_varpool_node_set,
> ? ? ? ?varpool_get_node, varpool_remove_node): Declare.
> ? ? ? ?(vsi_end_p, vsi_next, vsi_node, vsi_start, varpool_node_in_set_p,
> ? ? ? ?varpool_node_set_size): New inlines.
> ? ? ? ?* tree-pass.h (varpool_node_set_def): Forward declare.
> ? ? ? ?(ipa_opt_pass_d): Summary writting takes vnode sets too.
> ? ? ? ?(ipa_write_optimization_summaries): Update prototype.
> ? ? ? ?* ipa-cp.c (ipcp_write_summary): Update.
> ? ? ? ?* ipa-reference.c (ipa_reference_write_summary): Update.
> ? ? ? ?* lto-cgraph.c (lto_output_varpool_node): New static function.
> ? ? ? ?(output_varpool): New function.
> ? ? ? ?(input_varpool_node): New static function.
> ? ? ? ?(input_varpool_1): New function.
> ? ? ? ?(input_cgraph): Input varpool.
> ? ? ? ?* ipa-pure-const.c (pure_const_write_summary): Update.
> ? ? ? ?* lto-streamer-out.c (lto_output): Update, output varpool too.
> ? ? ? ?(write_global_stream): Kill WPA hack.
> ? ? ? ?(produce_asm_for_decls): Update.
> ? ? ? ?* ipa-inline.c (inline_write_summary): Update.
> ? ? ? ?* lto-streamer-in.c (lto_input_tree_ref, lto_input_tree): Do not build cgraph.
> ? ? ? ?* lto-section-in.c (lto_section_name): Add varpool and jump funcs.
> ? ? ? ?* ipa.c (hash_varpool_node_set_element, eq_varpool_node_set_element,
> ? ? ? ?varpool_node_set_new, varpool_node_set_add,
> ? ? ? ?varpool_node_set_remove, varpool_node_set_find, dump_varpool_node_set,
> ? ? ? ?debug_varpool_node_set): New functions.
> ? ? ? ?* passes.c (rest_of_decl_compilation): when in LTO do not finalize.
> ? ? ? ?(execute_one_pass): Process new decls too.
> ? ? ? ?(ipa_write_summaries_2): Pass around vsets.
> ? ? ? ?(ipa_write_summaries_1): Likewise.
> ? ? ? ?(ipa_write_summaries): Build vset; be more selective about cgraph nodes
> ? ? ? ?to add.
> ? ? ? ?(ipa_write_optimization_summaries_1): Pass around vsets.
> ? ? ? ?(ipa_write_optimization_summaries): Likewise.
> ? ? ? ?* varpool.c (varpool_get_node): New.
> ? ? ? ?(varpool_node): Update doubly linked lists.
> ? ? ? ?(varpool_remove_node): New.
> ? ? ? ?(dump_varpool_node): More dumping.
> ? ? ? ?(varpool_enqueue_needed_node): Update doubly linked lists.
> ? ? ? ?(decide_is_variable_needed): Kill ltrans hack.
> ? ? ? ?(varpool_finalize_decl): Kill lto hack.
> ? ? ? ?(varpool_assemble_decl): Skip decls in other partitions.
> ? ? ? ?(varpool_assemble_pending_decls): Update doubly linkes lists.
> ? ? ? ?(varpool_empty_needed_queue): Likewise.
> ? ? ? ?(varpool_extra_name_alias): Likewise.
> ? ? ? ?* lto-streamer.c (lto_get_section_name): Add vars section.
> ? ? ? ?* lto-streamer.h (lto_section_type): Update.
> ? ? ? ?(output_varpool, input_varpool): Declare.
>
> ? ? ? ?* lto.c (lto_varpool_node_sets): New.
> ? ? ? ?(lto_1_to_1_map): Partition varpool too.
> ? ? ? ?(globalize_context_t, globalize_cross_file_statics,
> ? ? ? ?lto_scan_statics_in_ref_table, lto_scan_statics_in_cgraph_node,
> ? ? ? ?lto_scan_statics_in_remaining_global_vars): Remove.
> ? ? ? ?(lto_promote_cross_file_statics): Rewrite.
> ? ? ? ?(get_filename_for_set): Take vset argument.
> ? ? ? ?(lto_wpa_write_files): Pass around vsets.
>

This caused:

http://gcc.gnu.org/bugzilla/show_bug.cgi?id=43938

-- 
H.J.