Re: More of ipa-inline housekeeping

[Richard Guenther <richard dot guenther at gmail dot com>]

On Thu, Apr 14, 2011 at 12:20 AM, Jan Hubicka <hubicka@ucw.cz> wrote:
> Hi,
> this patch moves inline_summary from field in cgraph_node into its own on side
> datastructure. This moves it from arcane decision of mine to split all IPA data
> into global/local datas stored in common datastructure into the scheme we
> developed for new IPA passes some time ago.
>
> The advantage is that the code is more contained and less spread across the
> compiler. We also make cgraph_node smaller and dumps more compact that never
> hurts.
>
> While working on it I noticed that Richi's patch to introduce cgraph_edge
> times/sizes is bit iffy in computing data when they are missing in the
> datastructure. Also it computes incomming edge costs instead of outgoing that
> leads to fact that not all edges gets their info computed for IPA inliner
> (think of newly discovered direct calls or IPA merging).

Ah, that was the reason ... I didn't dig deep enough ... ;)

>
> I fixed this on the and added sanity check that the fields are initialized.
> This has shown problem with early inliner iteration fixed thusly and fact that
> early inliner is attempting to compute overall growth at a time the inline
> parameters are not computed for functions not visited by early optimizations
> yet. We previously agreed that early inliner should not try to do that (as this
> leads to early inliner inlining functions called once that should be deferred
> for later consieration). ?I just hope it won't cause benchmarks to
> regress too much ;)

Yeah, we agreed to that.  And I forgot about it as it wasn't part of the
early inliner reorg (which was supposed to be a 1:1 transform).

>
> Having place to pile inline analysis info in, there is more to cleanup. The
> cgraph_local/cgraph_global fields probably should go and the stuff from global
> info should go into inline_summary datastructure, too (the lifetimes are
> essentially the same so there is no need for the split). ?I will handle this
> incrementally.
>
> Bootstrapped/regtested x86_64-linux with slightly modified version of the patch.
> Re-testing with final version and intend to commit the patch tomorrow.

I looked over the patch and it looks ok to me.

Thanks,
Richard.

> Honza
>
> ? ? ? ?* cgraph.c (dump_cgraph_node): Do not dump inline summaries.
> ? ? ? ?* cgraph.h (struct inline_summary): Move to ipa-inline.h
> ? ? ? ?(cgraph_local_info): Remove inline_summary.
> ? ? ? ?* ipa-cp.c: Include ipa-inline.h.
> ? ? ? ?(ipcp_cloning_candidate_p, ipcp_estimate_growth,
> ? ? ? ?ipcp_estimate_cloning_cost, ipcp_insert_stage): Use inline_summary
> ? ? ? ?accesor.
> ? ? ? ?* lto-cgraph.c (lto_output_node): Do not stream inline summary.
> ? ? ? ?(input_overwrite_node): Do not set inline summary.
> ? ? ? ?(input_node): Do not stream inline summary.
> ? ? ? ?* ipa-inline.c (cgraph_decide_inlining): Dump inline summaries.
> ? ? ? ?(cgraph_decide_inlining_incrementally): Do not try to estimate overall
> ? ? ? ?growth; we do not have inline parameters computed for that anyway.
> ? ? ? ?(cgraph_early_inlining): After inlining compute call_stmt_sizes.
> ? ? ? ?* ipa-inline.h (struct inline_summary): Move here from ipa-inline.h
> ? ? ? ?(inline_summary_t): New type and VECtor.
> ? ? ? ?(debug_inline_summary, dump_inline_summaries): Declare.
> ? ? ? ?(inline_summary): Use VOCtor.
> ? ? ? ?(estimate_edge_growth): Kill hack computing call stmt size directly.
> ? ? ? ?* lto-section-in.c (lto_section_name): Add inline section.
> ? ? ? ?* ipa-inline-analysis.c: Include lto-streamer.h
> ? ? ? ?(node_removal_hook_holder, node_duplication_hook_holder): New holders
> ? ? ? ?(inline_node_removal_hook, inline_node_duplication_hook): New functions.
> ? ? ? ?(inline_summary_vec): Define.
> ? ? ? ?(inline_summary_alloc, dump_inline_summary, debug_inline_summary,
> ? ? ? ?dump_inline_summaries): New functions.
> ? ? ? ?(estimate_function_body_sizes): Properly compute size/time of outgoing calls.
> ? ? ? ?(compute_inline_parameters): Alloc inline_summary; do not compute size/time
> ? ? ? ?of incomming calls.
> ? ? ? ?(estimate_edge_time): Avoid missing time summary hack.
> ? ? ? ?(inline_read_summary): Read inline summary info.
> ? ? ? ?(inline_write_summary): Write inline summary info.
> ? ? ? ?(inline_free_summary): Free all hooks and inline summary vector.
> ? ? ? ?* lto-streamer.h: Add LTO_section_inline_summary section.
> ? ? ? ?* Makefile.in (ipa-cp.o, ipa-inline-analysis.o): Update dependencies.
> ? ? ? ?* ipa.c (cgraph_remove_unreachable_nodes): Fix dump file formating.
>
> ? ? ? ?* lto.c: Include ipa-inline.h
> ? ? ? ?(add_cgraph_node_to_partition, undo_partition): Use inline_summary accessor.
> ? ? ? ?(ipa_node_duplication_hook): Fix declaration.
> ? ? ? ?* Make-lang.in (lto.o): Update dependencies.
> Index: cgraph.c
> ===================================================================
> --- cgraph.c ? ?(revision 172396)
> +++ cgraph.c ? ?(working copy)
> @@ -1876,22 +1876,6 @@ dump_cgraph_node (FILE *f, struct cgraph
> ? if (node->count)
> ? ? fprintf (f, " executed "HOST_WIDEST_INT_PRINT_DEC"x",
> ? ? ? ? ? ? (HOST_WIDEST_INT)node->count);
> - ?if (node->local.inline_summary.self_time)
> - ? ?fprintf (f, " %i time, %i benefit", node->local.inline_summary.self_time,
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? node->local.inline_summary.time_inlining_benefit);
> - ?if (node->global.time && node->global.time
> - ? ? ?!= node->local.inline_summary.self_time)
> - ? ?fprintf (f, " (%i after inlining)", node->global.time);
> - ?if (node->local.inline_summary.self_size)
> - ? ?fprintf (f, " %i size, %i benefit", node->local.inline_summary.self_size,
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? node->local.inline_summary.size_inlining_benefit);
> - ?if (node->global.size && node->global.size
> - ? ? ?!= node->local.inline_summary.self_size)
> - ? ?fprintf (f, " (%i after inlining)", node->global.size);
> - ?if (node->local.inline_summary.estimated_self_stack_size)
> - ? ?fprintf (f, " %i bytes stack usage", (int)node->local.inline_summary.estimated_self_stack_size);
> - ?if (node->global.estimated_stack_size != node->local.inline_summary.estimated_self_stack_size)
> - ? ?fprintf (f, " %i bytes after inlining", (int)node->global.estimated_stack_size);
> ? if (node->origin)
> ? ? fprintf (f, " nested in: %s", cgraph_node_name (node->origin));
> ? if (node->needed)
> Index: cgraph.h
> ===================================================================
> --- cgraph.h ? ?(revision 172396)
> +++ cgraph.h ? ?(working copy)
> @@ -58,23 +58,6 @@ struct lto_file_decl_data;
> ?extern const char * const cgraph_availability_names[];
> ?extern const char * const ld_plugin_symbol_resolution_names[];
>
> -/* Function inlining information. ?*/
> -
> -struct GTY(()) inline_summary
> -{
> - ?/* Estimated stack frame consumption by the function. ?*/
> - ?HOST_WIDE_INT estimated_self_stack_size;
> -
> - ?/* Size of the function body. ?*/
> - ?int self_size;
> - ?/* How many instructions are likely going to disappear after inlining. ?*/
> - ?int size_inlining_benefit;
> - ?/* Estimated time spent executing the function body. ?*/
> - ?int self_time;
> - ?/* How much time is going to be saved by inlining. ?*/
> - ?int time_inlining_benefit;
> -};
> -
> ?/* Information about thunk, used only for same body aliases. ?*/
>
> ?struct GTY(()) cgraph_thunk_info {
> @@ -95,8 +78,6 @@ struct GTY(()) cgraph_local_info {
> ? /* File stream where this node is being written to. ?*/
> ? struct lto_file_decl_data * lto_file_data;
>
> - ?struct inline_summary inline_summary;
> -
> ? /* Set when function function is visible in current compilation unit only
> ? ? ?and its address is never taken. ?*/
> ? unsigned local : 1;
> Index: ipa-cp.c
> ===================================================================
> --- ipa-cp.c ? ?(revision 172396)
> +++ ipa-cp.c ? ?(working copy)
> @@ -148,6 +148,7 @@ along with GCC; see the file COPYING3.
> ?#include "tree-inline.h"
> ?#include "fibheap.h"
> ?#include "params.h"
> +#include "ipa-inline.h"
>
> ?/* Number of functions identified as candidates for cloning. When not cloning
> ? ?we can simplify iterate stage not forcing it to go through the decision
> @@ -495,7 +496,7 @@ ipcp_cloning_candidate_p (struct cgraph_
> ? ? ? ? ? ? ? ? cgraph_node_name (node));
> ? ? ? return false;
> ? ? }
> - ?if (node->local.inline_summary.self_size < n_calls)
> + ?if (inline_summary (node)->self_size < n_calls)
> ? ? {
> ? ? ? if (dump_file)
> ? ? ? ? fprintf (dump_file, "Considering %s for cloning; code would shrink.\n",
> @@ -1189,7 +1190,7 @@ ipcp_estimate_growth (struct cgraph_node
> ? ? ?call site. ?Precise cost is difficult to get, as our size metric counts
> ? ? ?constants and moves as free. ?Generally we are looking for cases that
> ? ? ?small function is called very many times. ?*/
> - ?growth = node->local.inline_summary.self_size
> + ?growth = inline_summary (node)->self_size
> ? ? ? ? ? - removable_args * redirectable_node_callers;
> ? if (growth < 0)
> ? ? return 0;
> @@ -1229,7 +1230,7 @@ ipcp_estimate_cloning_cost (struct cgrap
> ? ? cost /= freq_sum * 1000 / REG_BR_PROB_BASE + 1;
> ? if (dump_file)
> ? ? fprintf (dump_file, "Cost of versioning %s is %i, (size: %i, freq: %i)\n",
> - ? ? ? ? ? ? cgraph_node_name (node), cost, node->local.inline_summary.self_size,
> + ? ? ? ? ? ? cgraph_node_name (node), cost, inline_summary (node)->self_size,
> ? ? ? ? ? ? freq_sum);
> ? return cost + 1;
> ?}
> @@ -1364,7 +1365,7 @@ ipcp_insert_stage (void)
> ? ? ? {
> ? ? ? ?if (node->count > max_count)
> ? ? ? ? ?max_count = node->count;
> - ? ? ? overall_size += node->local.inline_summary.self_size;
> + ? ? ? overall_size += inline_summary (node)->self_size;
> ? ? ? }
>
> ? max_new_size = overall_size;
> Index: lto-cgraph.c
> ===================================================================
> --- lto-cgraph.c ? ? ? ?(revision 172396)
> +++ lto-cgraph.c ? ? ? ?(working copy)
> @@ -465,16 +465,6 @@ lto_output_node (struct lto_simple_outpu
>
> ? if (tag == LTO_cgraph_analyzed_node)
> ? ? {
> - ? ? ?lto_output_sleb128_stream (ob->main_stream,
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?node->local.inline_summary.estimated_self_stack_size);
> - ? ? ?lto_output_sleb128_stream (ob->main_stream,
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?node->local.inline_summary.self_size);
> - ? ? ?lto_output_sleb128_stream (ob->main_stream,
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?node->local.inline_summary.size_inlining_benefit);
> - ? ? ?lto_output_sleb128_stream (ob->main_stream,
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?node->local.inline_summary.self_time);
> - ? ? ?lto_output_sleb128_stream (ob->main_stream,
> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?node->local.inline_summary.time_inlining_benefit);
> ? ? ? if (node->global.inlined_to)
> ? ? ? ?{
> ? ? ? ? ?ref = lto_cgraph_encoder_lookup (encoder, node->global.inlined_to);
> @@ -930,23 +920,9 @@ input_overwrite_node (struct lto_file_de
> ? ? ? ? ? ? ? ? ? ? ?struct cgraph_node *node,
> ? ? ? ? ? ? ? ? ? ? ?enum LTO_cgraph_tags tag,
> ? ? ? ? ? ? ? ? ? ? ?struct bitpack_d *bp,
> - ? ? ? ? ? ? ? ? ? ? unsigned int stack_size,
> - ? ? ? ? ? ? ? ? ? ? unsigned int self_time,
> - ? ? ? ? ? ? ? ? ? ? unsigned int time_inlining_benefit,
> - ? ? ? ? ? ? ? ? ? ? unsigned int self_size,
> - ? ? ? ? ? ? ? ? ? ? unsigned int size_inlining_benefit,
> ? ? ? ? ? ? ? ? ? ? ?enum ld_plugin_symbol_resolution resolution)
> ?{
> ? node->aux = (void *) tag;
> - ?node->local.inline_summary.estimated_self_stack_size = stack_size;
> - ?node->local.inline_summary.self_time = self_time;
> - ?node->local.inline_summary.time_inlining_benefit = time_inlining_benefit;
> - ?node->local.inline_summary.self_size = self_size;
> - ?node->local.inline_summary.size_inlining_benefit = size_inlining_benefit;
> - ?node->global.time = self_time;
> - ?node->global.size = self_size;
> - ?node->global.estimated_stack_size = stack_size;
> - ?node->global.estimated_growth = INT_MIN;
> ? node->local.lto_file_data = file_data;
>
> ? node->local.local = bp_unpack_value (bp, 1);
> @@ -1023,13 +999,8 @@ input_node (struct lto_file_decl_data *f
> ? tree fn_decl;
> ? struct cgraph_node *node;
> ? struct bitpack_d bp;
> - ?int stack_size = 0;
> ? unsigned decl_index;
> ? int ref = LCC_NOT_FOUND, ref2 = LCC_NOT_FOUND;
> - ?int self_time = 0;
> - ?int self_size = 0;
> - ?int time_inlining_benefit = 0;
> - ?int size_inlining_benefit = 0;
> ? unsigned long same_body_count = 0;
> ? int clone_ref;
> ? enum ld_plugin_symbol_resolution resolution;
> @@ -1051,15 +1022,7 @@ input_node (struct lto_file_decl_data *f
> ? node->count_materialization_scale = lto_input_sleb128 (ib);
>
> ? if (tag == LTO_cgraph_analyzed_node)
> - ? ?{
> - ? ? ?stack_size = lto_input_sleb128 (ib);
> - ? ? ?self_size = lto_input_sleb128 (ib);
> - ? ? ?size_inlining_benefit = lto_input_sleb128 (ib);
> - ? ? ?self_time = lto_input_sleb128 (ib);
> - ? ? ?time_inlining_benefit = lto_input_sleb128 (ib);
> -
> - ? ? ?ref = lto_input_sleb128 (ib);
> - ? ?}
> + ? ?ref = lto_input_sleb128 (ib);
>
> ? ref2 = lto_input_sleb128 (ib);
>
> @@ -1073,9 +1036,7 @@ input_node (struct lto_file_decl_data *f
>
> ? bp = lto_input_bitpack (ib);
> ? resolution = (enum ld_plugin_symbol_resolution)lto_input_uleb128 (ib);
> - ?input_overwrite_node (file_data, node, tag, &bp, stack_size, self_time,
> - ? ? ? ? ? ? ? ? ? ? ? time_inlining_benefit, self_size,
> - ? ? ? ? ? ? ? ? ? ? ? size_inlining_benefit, resolution);
> + ?input_overwrite_node (file_data, node, tag, &bp, resolution);
>
> ? /* Store a reference for now, and fix up later to be a pointer. ?*/
> ? node->global.inlined_to = (cgraph_node_ptr) (intptr_t) ref;
> Index: ipa-inline.c
> ===================================================================
> --- ipa-inline.c ? ? ? ?(revision 172396)
> +++ ipa-inline.c ? ? ? ?(working copy)
> @@ -1301,6 +1301,9 @@ cgraph_decide_inlining (void)
> ? ? ? ? ? ? ?max_benefit = benefit;
> ? ? ? ? ?}
> ? ? ? }
> +
> + ?if (dump_file)
> + ? ?dump_inline_summaries (dump_file);
> ? gcc_assert (in_lto_p
> ? ? ? ? ? ? ?|| !max_count
> ? ? ? ? ? ? ?|| (profile_info && flag_branch_probabilities));
> @@ -1558,8 +1561,7 @@ cgraph_decide_inlining_incrementally (st
> ? ? ? /* When the function body would grow and inlining the function
> ? ? ? ? won't eliminate the need for offline copy of the function,
> ? ? ? ? don't inline. ?*/
> - ? ? ?if (estimate_edge_growth (e) > allowed_growth
> - ? ? ? ? && estimate_growth (e->callee) > allowed_growth)
> + ? ? ?if (estimate_edge_growth (e) > allowed_growth)
> ? ? ? ?{
> ? ? ? ? ?if (dump_file)
> ? ? ? ? ? ?fprintf (dump_file,
> @@ -1601,6 +1603,7 @@ static unsigned int
> ?cgraph_early_inlining (void)
> ?{
> ? struct cgraph_node *node = cgraph_get_node (current_function_decl);
> + ?struct cgraph_edge *edge;
> ? unsigned int todo = 0;
> ? int iterations = 0;
> ? bool inlined = false;
> @@ -1652,6 +1655,19 @@ cgraph_early_inlining (void)
> ? ? {
> ? ? ? timevar_push (TV_INTEGRATION);
> ? ? ? todo |= optimize_inline_calls (current_function_decl);
> +
> + ? ? ?/* Technically we ought to recompute inline parameters so the new iteration of
> + ? ? ? ?early inliner works as expected. ?We however have values approximately right
> + ? ? ? ?and thus we only need to update edge info that might be cleared out for
> + ? ? ? ?newly discovered edges. ?*/
> + ? ? ?for (edge = node->callees; edge; edge = edge->next_callee)
> + ? ? ? {
> + ? ? ? ? edge->call_stmt_size
> + ? ? ? ? ? = estimate_num_insns (edge->call_stmt, &eni_size_weights);
> + ? ? ? ? edge->call_stmt_time
> + ? ? ? ? ? = estimate_num_insns (edge->call_stmt, &eni_time_weights);
> + ? ? ? }
> +
> ? ? ? timevar_pop (TV_INTEGRATION);
> ? ? }
>
> Index: ipa-inline.h
> ===================================================================
> --- ipa-inline.h ? ? ? ?(revision 172396)
> +++ ipa-inline.h ? ? ? ?(working copy)
> @@ -19,6 +19,30 @@ You should have received a copy of the G
> ?along with GCC; see the file COPYING3. ?If not see
> ?<http://www.gnu.org/licenses/>. ?*/
>
> +/* Function inlining information. ?*/
> +
> +struct inline_summary
> +{
> + ?/* Estimated stack frame consumption by the function. ?*/
> + ?HOST_WIDE_INT estimated_self_stack_size;
> +
> + ?/* Size of the function body. ?*/
> + ?int self_size;
> + ?/* How many instructions are likely going to disappear after inlining. ?*/
> + ?int size_inlining_benefit;
> + ?/* Estimated time spent executing the function body. ?*/
> + ?int self_time;
> + ?/* How much time is going to be saved by inlining. ?*/
> + ?int time_inlining_benefit;
> +};
> +
> +typedef struct inline_summary inline_summary_t;
> +DEF_VEC_O(inline_summary_t);
> +DEF_VEC_ALLOC_O(inline_summary_t,heap);
> +extern VEC(inline_summary_t,heap) *inline_summary_vec;
> +
> +void debug_inline_summary (struct cgraph_node *);
> +void dump_inline_summaries (FILE *f);
> ?void inline_generate_summary (void);
> ?void inline_read_summary (void);
> ?void inline_write_summary (cgraph_node_set, varpool_node_set);
> @@ -30,7 +54,7 @@ int estimate_growth (struct cgraph_node
> ?static inline struct inline_summary *
> ?inline_summary (struct cgraph_node *node)
> ?{
> - ?return &node->local.inline_summary;
> + ?return VEC_index (inline_summary_t, inline_summary_vec, node->uid);
> ?}
>
> ?/* Estimate the growth of the caller when inlining EDGE. ?*/
> @@ -39,12 +63,8 @@ static inline int
> ?estimate_edge_growth (struct cgraph_edge *edge)
> ?{
> ? int call_stmt_size;
> - ?/* ??? ?We throw away cgraph edges all the time so the information
> - ? ? we store in edges doesn't persist for early inlining. ?Ugh. ?*/
> - ?if (!edge->call_stmt)
> - ? ?call_stmt_size = edge->call_stmt_size;
> - ?else
> - ? ?call_stmt_size = estimate_num_insns (edge->call_stmt, &eni_size_weights);
> + ?call_stmt_size = edge->call_stmt_size;
> + ?gcc_checking_assert (call_stmt_size);
> ? return (edge->callee->global.size
> ? ? ? ? ?- inline_summary (edge->callee)->size_inlining_benefit
> ? ? ? ? ?- call_stmt_size);
> Index: lto-section-in.c
> ===================================================================
> --- lto-section-in.c ? ?(revision 172396)
> +++ lto-section-in.c ? ?(working copy)
> @@ -58,7 +58,8 @@ const char *lto_section_name[LTO_N_SECTI
> ? "reference",
> ? "symtab",
> ? "opts",
> - ?"cgraphopt"
> + ?"cgraphopt",
> + ?"inline"
> ?};
>
> ?unsigned char
> Index: ipa.c
> ===================================================================
> --- ipa.c ? ? ? (revision 172396)
> +++ ipa.c ? ? ? (working copy)
> @@ -517,6 +517,8 @@ cgraph_remove_unreachable_nodes (bool be
> ? ? ? ? ? ? ?}
> ? ? ? ? ?}
> ? ? ? }
> + ?if (file)
> + ? ?fprintf (file, "\n");
>
> ?#ifdef ENABLE_CHECKING
> ? verify_cgraph ();
> Index: ipa-inline-analysis.c
> ===================================================================
> --- ipa-inline-analysis.c ? ? ? (revision 172396)
> +++ ipa-inline-analysis.c ? ? ? (working copy)
> @@ -23,13 +23,13 @@ along with GCC; see the file COPYING3.
>
> ? ?We estimate for each function
> ? ? ?- function body size
> - ? ? - function runtime
> + ? ? - average function execution time
> ? ? ?- inlining size benefit (that is how much of function body size
> ? ? ? ?and its call sequence is expected to disappear by inlining)
> ? ? ?- inlining time benefit
> ? ? ?- function frame size
> ? ?For each call
> - ? ? - call sequence size
> + ? ? - call statement size and time
>
> ? ?inlinie_summary datastructures store above information locally (i.e.
> ? ?parameters of the function itself) and globally (i.e. parameters of
> @@ -61,12 +61,100 @@ along with GCC; see the file COPYING3.
> ?#include "ggc.h"
> ?#include "tree-flow.h"
> ?#include "ipa-prop.h"
> +#include "lto-streamer.h"
> ?#include "ipa-inline.h"
>
> ?#define MAX_TIME 1000000000
>
> ?/* Holders of ipa cgraph hooks: */
> ?static struct cgraph_node_hook_list *function_insertion_hook_holder;
> +static struct cgraph_node_hook_list *node_removal_hook_holder;
> +static struct cgraph_2node_hook_list *node_duplication_hook_holder;
> +static void inline_node_removal_hook (struct cgraph_node *, void *);
> +static void inline_node_duplication_hook (struct cgraph_node *,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? struct cgraph_node *, void *);
> +
> +/* VECtor holding inline summaries. ?*/
> +VEC(inline_summary_t,heap) *inline_summary_vec;
> +
> +/* Allocate the inline summary vector or resize it to cover all cgraph nodes. */
> +
> +static void
> +inline_summary_alloc (void)
> +{
> + ?if (!node_removal_hook_holder)
> + ? ?node_removal_hook_holder =
> + ? ? ?cgraph_add_node_removal_hook (&inline_node_removal_hook, NULL);
> + ?if (!node_duplication_hook_holder)
> + ? ?node_duplication_hook_holder =
> + ? ? ?cgraph_add_node_duplication_hook (&inline_node_duplication_hook, NULL);
> +
> + ?if (VEC_length (inline_summary_t, inline_summary_vec)
> + ? ? ?<= (unsigned) cgraph_max_uid)
> + ? ?VEC_safe_grow_cleared (inline_summary_t, heap,
> + ? ? ? ? ? ? ? ? ? ? ? ? ?inline_summary_vec, cgraph_max_uid + 1);
> +}
> +
> +/* Hook that is called by cgraph.c when a node is removed. ?*/
> +
> +static void
> +inline_node_removal_hook (struct cgraph_node *node, void *data ATTRIBUTE_UNUSED)
> +{
> + ?/* During IPA-CP updating we can be called on not-yet analyze clones. ?*/
> + ?if (VEC_length (inline_summary_t, inline_summary_vec)
> + ? ? ?<= (unsigned)node->uid)
> + ? ?return;
> + ?memset (inline_summary (node),
> + ? ? ? ? 0, sizeof (inline_summary_t));
> +}
> +
> +/* Hook that is called by cgraph.c when a node is duplicated. ?*/
> +
> +static void
> +inline_node_duplication_hook (struct cgraph_node *src, struct cgraph_node *dst,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ATTRIBUTE_UNUSED void *data)
> +{
> + ?inline_summary_alloc ();
> + ?memcpy (inline_summary (dst), inline_summary (src),
> + ? ? ? ? sizeof (struct inline_summary));
> +}
> +
> +static void
> +dump_inline_summary (FILE *f, struct cgraph_node *node)
> +{
> + ?if (node->analyzed)
> + ? ?{
> + ? ? ?struct inline_summary *s = inline_summary (node);
> + ? ? ?fprintf (f, "Inline summary for %s/%i\n", cgraph_node_name (node),
> + ? ? ? ? ? ? ?node->uid);
> + ? ? ?fprintf (f, " ?self time: ? ? ? %i, benefit: %i\n",
> + ? ? ? ? ? ? ?s->self_time, s->time_inlining_benefit);
> + ? ? ?fprintf (f, " ?global time: ? ? %i\n", node->global.time);
> + ? ? ?fprintf (f, " ?self size: ? ? ? %i, benefit: %i\n",
> + ? ? ? ? ? ? ?s->self_size, s->size_inlining_benefit);
> + ? ? ?fprintf (f, " ?global size: ? ? %i", node->global.size);
> + ? ? ?fprintf (f, " ?self stack: ? ? ?%i\n",
> + ? ? ? ? ? ? ?(int)s->estimated_self_stack_size);
> + ? ? ?fprintf (f, " ?global stack: ? ?%i\n",
> + ? ? ? ? ? ? ?(int)node->global.estimated_stack_size);
> + ? ?}
> +}
> +
> +void
> +debug_inline_summary (struct cgraph_node *node)
> +{
> + ?dump_inline_summary (stderr, node);
> +}
> +
> +void
> +dump_inline_summaries (FILE *f)
> +{
> + ?struct cgraph_node *node;
> +
> + ?for (node = cgraph_nodes; node; node = node->next)
> + ? ?if (node->analyzed)
> + ? ? ?dump_inline_summary (f, node);
> +}
>
> ?/* See if statement might disappear after inlining.
> ? ?0 - means not eliminated
> @@ -179,16 +267,27 @@ estimate_function_body_sizes (struct cgr
> ? ? ? ? ? ? ? ? ? ? ? freq, this_size, this_time);
> ? ? ? ? ? ? ?print_gimple_stmt (dump_file, stmt, 0, 0);
> ? ? ? ? ? ?}
> +
> + ? ? ? ? if (is_gimple_call (stmt))
> + ? ? ? ? ? {
> + ? ? ? ? ? ? struct cgraph_edge *edge = cgraph_edge (node, stmt);
> + ? ? ? ? ? ? edge->call_stmt_size = this_size;
> + ? ? ? ? ? ? edge->call_stmt_time = this_time;
> + ? ? ? ? ? }
> +
> ? ? ? ? ?this_time *= freq;
> ? ? ? ? ?time += this_time;
> ? ? ? ? ?size += this_size;
> +
> ? ? ? ? ?prob = eliminated_by_inlining_prob (stmt);
> ? ? ? ? ?if (prob == 1 && dump_file && (dump_flags & TDF_DETAILS))
> ? ? ? ? ? ?fprintf (dump_file, " ? ?50%% will be eliminated by inlining\n");
> ? ? ? ? ?if (prob == 2 && dump_file && (dump_flags & TDF_DETAILS))
> ? ? ? ? ? ?fprintf (dump_file, " ? ?will eliminated by inlining\n");
> +
> ? ? ? ? ?size_inlining_benefit += this_size * prob;
> ? ? ? ? ?time_inlining_benefit += this_time * prob;
> +
> ? ? ? ? ?gcc_assert (time >= 0);
> ? ? ? ? ?gcc_assert (size >= 0);
> ? ? ? ?}
> @@ -222,6 +321,8 @@ compute_inline_parameters (struct cgraph
>
> ? gcc_assert (!node->global.inlined_to);
>
> + ?inline_summary_alloc ();
> +
> ? /* Estimate the stack size for the function if we're optimizing. ?*/
> ? self_stack_size = optimize ? estimated_stack_frame_size (node) : 0;
> ? inline_summary (node)->estimated_self_stack_size = self_stack_size;
> @@ -247,17 +348,7 @@ compute_inline_parameters (struct cgraph
> ? ? ? node->local.can_change_signature = !e;
> ? ? }
> ? estimate_function_body_sizes (node);
> - ?/* Compute size of call statements. ?We have to do this for callers here,
> - ? ? those sizes need to be present for edges _to_ us as early as
> - ? ? we are finished with early opts. ?*/
> - ?for (e = node->callers; e; e = e->next_caller)
> - ? ?if (e->call_stmt)
> - ? ? ?{
> - ? ? ? e->call_stmt_size
> - ? ? ? ? = estimate_num_insns (e->call_stmt, &eni_size_weights);
> - ? ? ? e->call_stmt_time
> - ? ? ? ? = estimate_num_insns (e->call_stmt, &eni_time_weights);
> - ? ? ?}
> +
> ? /* Inlining characteristics are maintained by the cgraph_mark_inline. ?*/
> ? node->global.time = inline_summary (node)->self_time;
> ? node->global.size = inline_summary (node)->self_size;
> @@ -300,12 +391,8 @@ static inline int
> ?estimate_edge_time (struct cgraph_edge *edge)
> ?{
> ? int call_stmt_time;
> - ?/* ??? ?We throw away cgraph edges all the time so the information
> - ? ? we store in edges doesn't persist for early inlining. ?Ugh. ?*/
> - ?if (!edge->call_stmt)
> - ? ?call_stmt_time = edge->call_stmt_time;
> - ?else
> - ? ?call_stmt_time = estimate_num_insns (edge->call_stmt, &eni_time_weights);
> + ?call_stmt_time = edge->call_stmt_time;
> + ?gcc_checking_assert (call_stmt_time);
> ? return (((gcov_type)edge->callee->global.time
> ? ? ? ? ? - inline_summary (edge->callee)->time_inlining_benefit
> ? ? ? ? ? - call_stmt_time) * edge->frequency
> @@ -379,8 +466,10 @@ estimate_growth (struct cgraph_node *nod
> ? return growth;
> ?}
>
> +
> ?/* This function performs intraprocedural analysis in NODE that is required to
> ? ?inline indirect calls. ?*/
> +
> ?static void
> ?inline_indirect_intraprocedural_analysis (struct cgraph_node *node)
> ?{
> @@ -437,8 +526,6 @@ inline_generate_summary (void)
> ? for (node = cgraph_nodes; node; node = node->next)
> ? ? if (node->analyzed)
> ? ? ? inline_analyze_function (node);
> -
> - ?return;
> ?}
>
>
> @@ -449,6 +536,57 @@ inline_generate_summary (void)
> ?void
> ?inline_read_summary (void)
> ?{
> + ?struct lto_file_decl_data **file_data_vec = lto_get_file_decl_data ();
> + ?struct lto_file_decl_data *file_data;
> + ?unsigned int j = 0;
> +
> + ?inline_summary_alloc ();
> +
> + ?while ((file_data = file_data_vec[j++]))
> + ? ?{
> + ? ? ?size_t len;
> + ? ? ?const char *data = lto_get_section_data (file_data, LTO_section_inline_summary, NULL, &len);
> +
> + ? ? ?struct lto_input_block *ib
> + ? ? ? = lto_create_simple_input_block (file_data,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?LTO_section_inline_summary,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?&data, &len);
> + ? ? ?if (ib)
> + ? ? ? {
> + ? ? ? ? unsigned int i;
> + ? ? ? ? unsigned int f_count = lto_input_uleb128 (ib);
> +
> + ? ? ? ? for (i = 0; i < f_count; i++)
> + ? ? ? ? ? {
> + ? ? ? ? ? ? unsigned int index;
> + ? ? ? ? ? ? struct cgraph_node *node;
> + ? ? ? ? ? ? struct inline_summary *info;
> + ? ? ? ? ? ? lto_cgraph_encoder_t encoder;
> +
> + ? ? ? ? ? ? index = lto_input_uleb128 (ib);
> + ? ? ? ? ? ? encoder = file_data->cgraph_node_encoder;
> + ? ? ? ? ? ? node = lto_cgraph_encoder_deref (encoder, index);
> + ? ? ? ? ? ? info = inline_summary (node);
> +
> + ? ? ? ? ? ? node->global.estimated_stack_size
> + ? ? ? ? ? ? ? = info->estimated_self_stack_size = lto_input_uleb128 (ib);
> + ? ? ? ? ? ? node->global.time = info->self_time = lto_input_uleb128 (ib);
> + ? ? ? ? ? ? info->time_inlining_benefit = lto_input_uleb128 (ib);
> + ? ? ? ? ? ? node->global.size = info->self_size = lto_input_uleb128 (ib);
> + ? ? ? ? ? ? info->size_inlining_benefit = lto_input_uleb128 (ib);
> + ? ? ? ? ? ? node->global.estimated_growth = INT_MIN;
> + ? ? ? ? ? }
> +
> + ? ? ? ? lto_destroy_simple_input_block (file_data,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? LTO_section_inline_summary,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ib, data, len);
> + ? ? ? }
> + ? ? ?else
> + ? ? ? /* Fatal error here. ?We do not want to support compiling ltrans units with
> + ? ? ? ? ?different version of compiler or different flags than the WPA unit, so
> + ? ? ? ? ?this should never happen. ?*/
> + ? ? ? fatal_error ("ipa reference summary is missing in ltrans unit");
> + ? ?}
> ? if (flag_indirect_inlining)
> ? ? {
> ? ? ? ipa_register_cgraph_hooks ();
> @@ -468,14 +606,57 @@ void
> ?inline_write_summary (cgraph_node_set set,
> ? ? ? ? ? ? ? ? ? ? ?varpool_node_set vset ATTRIBUTE_UNUSED)
> ?{
> + ?struct cgraph_node *node;
> + ?struct lto_simple_output_block *ob
> + ? ?= lto_create_simple_output_block (LTO_section_inline_summary);
> + ?lto_cgraph_encoder_t encoder = ob->decl_state->cgraph_node_encoder;
> + ?unsigned int count = 0;
> + ?int i;
> +
> + ?for (i = 0; i < lto_cgraph_encoder_size (encoder); i++)
> + ? ?if (lto_cgraph_encoder_deref (encoder, i)->analyzed)
> + ? ? ?count++;
> + ?lto_output_uleb128_stream (ob->main_stream, count);
> +
> + ?for (i = 0; i < lto_cgraph_encoder_size (encoder); i++)
> + ? ?{
> + ? ? ?node = lto_cgraph_encoder_deref (encoder, i);
> + ? ? ?if (node->analyzed)
> + ? ? ? {
> + ? ? ? ? struct inline_summary *info = inline_summary (node);
> + ? ? ? ? lto_output_uleb128_stream (ob->main_stream,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?lto_cgraph_encoder_encode (encoder, node));
> + ? ? ? ? lto_output_sleb128_stream (ob->main_stream,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?info->estimated_self_stack_size);
> + ? ? ? ? lto_output_sleb128_stream (ob->main_stream,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?info->self_size);
> + ? ? ? ? lto_output_sleb128_stream (ob->main_stream,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?info->size_inlining_benefit);
> + ? ? ? ? lto_output_sleb128_stream (ob->main_stream,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?info->self_time);
> + ? ? ? ? lto_output_sleb128_stream (ob->main_stream,
> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?info->time_inlining_benefit);
> + ? ? ? }
> + ? ?}
> +
> ? if (flag_indirect_inlining && !flag_ipa_cp)
> ? ? ipa_prop_write_jump_functions (set);
> ?}
>
> +
> ?/* Release inline summary. ?*/
>
> ?void
> ?inline_free_summary (void)
> ?{
> - ?cgraph_remove_function_insertion_hook (function_insertion_hook_holder);
> + ?if (function_insertion_hook_holder)
> + ? ?cgraph_remove_function_insertion_hook (function_insertion_hook_holder);
> + ?function_insertion_hook_holder = NULL;
> + ?if (node_removal_hook_holder)
> + ? ?cgraph_remove_node_removal_hook (node_removal_hook_holder);
> + ?node_removal_hook_holder = NULL;
> + ?if (node_duplication_hook_holder)
> + ? ?cgraph_remove_node_duplication_hook (node_duplication_hook_holder);
> + ?node_duplication_hook_holder = NULL;
> + ?VEC_free (inline_summary_t, heap, inline_summary_vec);
> ?}
> Index: lto/lto.c
> ===================================================================
> --- lto/lto.c ? (revision 172396)
> +++ lto/lto.c ? (working copy)
> @@ -44,6 +44,7 @@ along with GCC; see the file COPYING3.
> ?#include "lto-streamer.h"
> ?#include "splay-tree.h"
> ?#include "params.h"
> +#include "ipa-inline.h"
>
> ?static GTY(()) tree first_personality_decl;
>
> @@ -750,7 +751,7 @@ add_cgraph_node_to_partition (ltrans_par
> ?{
> ? struct cgraph_edge *e;
>
> - ?part->insns += node->local.inline_summary.self_size;
> + ?part->insns += inline_summary (node)->self_size;
>
> ? if (node->aux)
> ? ? {
> @@ -811,7 +812,7 @@ undo_partition (ltrans_partition partiti
> ? ? ? struct cgraph_node *node = VEC_index (cgraph_node_ptr,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?partition->cgraph_set->nodes,
> ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?n_cgraph_nodes);
> - ? ? ?partition->insns -= node->local.inline_summary.self_size;
> + ? ? ?partition->insns -= inline_summary (node)->self_size;
> ? ? ? cgraph_node_set_remove (partition->cgraph_set, node);
> ? ? ? node->aux = (void *)((size_t)node->aux - 1);
> ? ? }
> Index: lto/Make-lang.in
> ===================================================================
> --- lto/Make-lang.in ? ?(revision 172396)
> +++ lto/Make-lang.in ? ?(working copy)
> @@ -85,7 +85,8 @@ lto/lto.o: lto/lto.c $(CONFIG_H) $(SYSTE
> ? ? ? ?$(CGRAPH_H) $(GGC_H) tree-ssa-operands.h $(TREE_PASS_H) \
> ? ? ? ?langhooks.h $(VEC_H) $(BITMAP_H) pointer-set.h $(IPA_PROP_H) \
> ? ? ? ?$(COMMON_H) debug.h $(TIMEVAR_H) $(GIMPLE_H) $(LTO_H) $(LTO_TREE_H) \
> - ? ? ? $(LTO_TAGS_H) $(LTO_STREAMER_H) $(SPLAY_TREE_H) gt-lto-lto.h $(PARAMS_H)
> + ? ? ? $(LTO_TAGS_H) $(LTO_STREAMER_H) $(SPLAY_TREE_H) gt-lto-lto.h $(PARAMS_H) \
> + ? ? ? ipa-inline.h
> ?lto/lto-object.o: lto/lto-object.c $(CONFIG_H) $(SYSTEM_H) coretypes.h \
> ? ? ? ?$(DIAGNOSTIC_CORE_H) $(LTO_H) $(TM_H) $(LTO_STREAMER_H) \
> ? ? ? ?../include/simple-object.h
> Index: ipa-prop.c
> ===================================================================
> --- ipa-prop.c ?(revision 172396)
> +++ ipa-prop.c ?(working copy)
> @@ -1998,7 +1998,7 @@ ipa_edge_duplication_hook (struct cgraph
>
> ?static void
> ?ipa_node_duplication_hook (struct cgraph_node *src, struct cgraph_node *dst,
> - ? ? ? ? ? ? ? ? ? ? ? ? ?__attribute__((unused)) void *data)
> + ? ? ? ? ? ? ? ? ? ? ? ? ?ATTRIBUTE_UNUSED void *data)
> ?{
> ? struct ipa_node_params *old_info, *new_info;
> ? int param_count, i;
> Index: Makefile.in
> ===================================================================
> --- Makefile.in (revision 172396)
> +++ Makefile.in (working copy)
> @@ -3011,7 +3011,7 @@ ipa-ref.o : ipa-ref.c $(CONFIG_H) $(SYST
> ?ipa-cp.o : ipa-cp.c $(CONFIG_H) $(SYSTEM_H) coretypes.h ?\
> ? ?$(TREE_H) $(TARGET_H) $(GIMPLE_H) $(CGRAPH_H) $(IPA_PROP_H) $(TREE_FLOW_H) \
> ? ?$(TREE_PASS_H) $(FLAGS_H) $(TIMEVAR_H) $(DIAGNOSTIC_H) $(TREE_DUMP_H) \
> - ? $(TREE_INLINE_H) $(FIBHEAP_H) $(PARAMS_H) tree-pretty-print.h
> + ? $(TREE_INLINE_H) $(FIBHEAP_H) $(PARAMS_H) tree-pretty-print.h ipa-inline.h
> ?ipa-split.o : ipa-split.c $(CONFIG_H) $(SYSTEM_H) coretypes.h ?\
> ? ?$(TREE_H) $(TARGET_H) $(CGRAPH_H) $(IPA_PROP_H) $(TREE_FLOW_H) \
> ? ?$(TREE_PASS_H) $(FLAGS_H) $(TIMEVAR_H) $(DIAGNOSTIC_H) $(TREE_DUMP_H) \
> @@ -3032,7 +3032,7 @@ ipa-inline-analysis.o : ipa-inline-analy
> ? ?$(TREE_H) langhooks.h $(TREE_INLINE_H) $(FLAGS_H) $(CGRAPH_H) intl.h \
> ? ?$(DIAGNOSTIC_H) $(PARAMS_H) $(TIMEVAR_H) $(TREE_PASS_H) \
> ? ?$(HASHTAB_H) $(COVERAGE_H) $(GGC_H) $(TREE_FLOW_H) $(IPA_PROP_H) \
> - ? gimple-pretty-print.h ipa-inline.h
> + ? gimple-pretty-print.h ipa-inline.h $(LTO_STREAMER_H)
> ?ipa-utils.o : ipa-utils.c $(IPA_UTILS_H) $(CONFIG_H) $(SYSTEM_H) \
> ? ?coretypes.h $(TM_H) $(TREE_H) $(TREE_FLOW_H) $(TREE_INLINE_H) langhooks.h \
> ? ?pointer-set.h $(GGC_H) $(GIMPLE_H) $(SPLAY_TREE_H) \
> Index: lto-streamer.h
> ===================================================================
> --- lto-streamer.h ? ? ?(revision 172396)
> +++ lto-streamer.h ? ? ?(working copy)
> @@ -264,6 +264,7 @@ enum lto_section_type
> ? LTO_section_symtab,
> ? LTO_section_opts,
> ? LTO_section_cgraph_opt_sum,
> + ?LTO_section_inline_summary,
> ? LTO_N_SECTION_TYPES ? ? ? ? ?/* Must be last. ?*/
> ?};
>
>