This is the mail archive of the
gcc-patches@gcc.gnu.org
mailing list for the GCC project.
Even more of inliner housekeeping
- From: Jan Hubicka <hubicka at ucw dot cz>
- To: gcc-patches at gcc dot gnu dot org
- Date: Sat, 16 Apr 2011 11:11:03 +0200
- Subject: Even more of inliner housekeeping
Hi,
this patch moves the inlining related fields from cgraph_local_info and
cgraph_global_info into the inline summaries.
It is largery mechanical patch with few exceptions. First "inlinable" flag
was tested at several times before it was computed. This is nonsense, but now
it also leads to range error on inline_summary_vec access.
I fixed this by removing those tests (that was there to initialize
inline_failed reason when function is not inlinable) and moved logic setting
CIF_FUNCTION_NOT_INLNABLE into inliner.
Second problem is with disregard_inline_limits. It is computed ahead of other
inlining parameters because of Richard's patch changing topological order to
ignore always inline functions (to allow inlining all of them early). This
flag largely duplicate flag DECL_DISREGARD_INLINE_LIMITS so I just moved
non-inliner uses to it.
In inliner I still use disregard_inline_limits since it is
DECL_DISREGARD_INLINE_LIMITS && inlinable. We might move away from it, but we
also might move to IPA passes without actual decls, so I don't know if we want
to remove this flag or add more similar flags and thus I opted to leave it
there for time being.
the "global.inlined_to" pointer is left in callgraph. It is more an representation
of inline plan that is part of virtual clones than an inliner own data.
As followup I plan to dismantle global/local substructures of cgraph_node and I
hope that datastructure wise I should be set with inliner.
Bootstrapped/regtested x86_64-linux, will commit it shortly.
Honza
* cgraphbuild.c: Include ipa-inline.h.
(reset_inline_failed): Use initialize_inline_failed.
* cgraph.c: Include ipa-inline.h.
(cgraph_create_node_1): Do not initialize estimated_growth.
(initialize_inline_failed): More to ipa-inline-analysis.c
(dump_cgraph_node): Do not dump inline flags.
* cgraph.h (cgraph_local_info): Remove inlineable, versionable
and disregard_inline_limits flags.
(cgrpah_global_info): Remove estimated_stack_size, stack_frame_offset,
time, size, estimated_growth.
* ipa-cp.c (ipcp_versionable_function_p, ipcp_generate_summary): Update.
* cgraphunit.c (cgraph_decide_is_function_needed): Use
DECL_DISREGARD_INLINE_LIMITS.
(cgraph_analyze_function): Do not initialize
node->local.disregard_inline_limits.
* lto-cgraph.c (lto_output_node, input_overwrite_node): Do not stream
inlinable, versionable and disregard_inline_limits.
* ipa-inline.c (cgraph_clone_inlined_nodes, cgraph_mark_inline_edge,
cgraph_check_inline_limits, cgraph_default_inline_p, cgraph_edge_badness,
update_caller_keys, update_callee_keys, add_new_edges_to_heap): Update.
(cgraph_decide_inlining_of_small_function): Update; set CIF_FUNCTION_NOT_INLINABLE
for uninlinable functions.
(cgraph_decide_inlining, cgraph_edge_early_inlinable_p,
cgraph_decide_inlining_incrementally): Update.
* ipa-inline.h (inline_summary): Add inlinable, versionable, disregard_inline_limits,
estimated_stack_size, stack_frame_offset, time, size and estimated_growth
parameters.
(estimate_edge_growth): Update.
(initialize_inline_failed): Declare.
* ipa-split.c: Include ipa-inline.h
(execute_split_functions): Update.
* ipa.c (cgraph_postorder): Use DECL_DISREGARD_INLINE_LIMITS.
(cgraph_remove_unreachable_nodes): Do not clear inlinable flag.
(record_cdtor_fn): Use DECL_DISREGARD_INLINE_LIMITS.
* ipa-inline-analysis.c (inline_node_removal_hook): Update; set
estimated_growth to INT_MIN.
(inline_node_duplication_hook): Likewise.
(dump_inline_summary): Dump new fields.
(compute_inline_parameters): Update.
(estimate_edge_time, estimate_time_after_inlining,
estimate_size_after_inlining, estimate_growth, inline_read_summary,
inline_write_summary):
(initialize_inline_failed): Move here from cgraph.c.
* tree-sra.c: Include ipa-inline.h.
(ipa_sra_preliminary_function_checks): Update.
* lto/lto.c (lto_balanced_map): Update.
Update.
* Makefile.in: (cgraph.o, cgraphbuild.o): Add dependency on
ipa-inline.h
Index: cgraphbuild.c
===================================================================
*** cgraphbuild.c (revision 172431)
--- cgraphbuild.c (working copy)
*************** along with GCC; see the file COPYING3.
*** 33,38 ****
--- 33,39 ----
#include "tree-pass.h"
#include "ipa-utils.h"
#include "except.h"
+ #include "ipa-inline.h"
/* Context of record_reference. */
struct record_reference_ctx
*************** reset_inline_failed (struct cgraph_node
*** 207,222 ****
for (e = node->callers; e; e = e->next_caller)
{
e->callee->global.inlined_to = NULL;
! if (!node->analyzed)
! e->inline_failed = CIF_BODY_NOT_AVAILABLE;
! else if (node->local.redefined_extern_inline)
! e->inline_failed = CIF_REDEFINED_EXTERN_INLINE;
! else if (!node->local.inlinable)
! e->inline_failed = CIF_FUNCTION_NOT_INLINABLE;
! else if (e->call_stmt_cannot_inline_p)
! e->inline_failed = CIF_MISMATCHED_ARGUMENTS;
! else
! e->inline_failed = CIF_FUNCTION_NOT_CONSIDERED;
}
}
--- 208,214 ----
for (e = node->callers; e; e = e->next_caller)
{
e->callee->global.inlined_to = NULL;
! initialize_inline_failed (e);
}
}
Index: cgraph.c
===================================================================
*** cgraph.c (revision 172431)
--- cgraph.c (working copy)
*************** The callgraph:
*** 98,103 ****
--- 98,104 ----
#include "rtl.h"
#include "ipa-utils.h"
#include "lto-streamer.h"
+ #include "ipa-inline.h"
const char * const ld_plugin_symbol_resolution_names[]=
{
*************** cgraph_create_node_1 (void)
*** 476,482 ****
if (cgraph_nodes)
cgraph_nodes->previous = node;
node->previous = NULL;
- node->global.estimated_growth = INT_MIN;
node->frequency = NODE_FREQUENCY_NORMAL;
node->count_materialization_scale = REG_BR_PROB_BASE;
ipa_empty_ref_list (&node->ref_list);
--- 477,482 ----
*************** cgraph_create_edge_including_clones (str
*** 970,997 ****
}
}
- /* Give initial reasons why inlining would fail on EDGE. This gets either
- nullified or usually overwritten by more precise reasons later. */
-
- static void
- initialize_inline_failed (struct cgraph_edge *e)
- {
- struct cgraph_node *callee = e->callee;
-
- if (e->indirect_unknown_callee)
- e->inline_failed = CIF_INDIRECT_UNKNOWN_CALL;
- else if (!callee->analyzed)
- e->inline_failed = CIF_BODY_NOT_AVAILABLE;
- else if (callee->local.redefined_extern_inline)
- e->inline_failed = CIF_REDEFINED_EXTERN_INLINE;
- else if (!callee->local.inlinable)
- e->inline_failed = CIF_FUNCTION_NOT_INLINABLE;
- else if (e->call_stmt && gimple_call_cannot_inline_p (e->call_stmt))
- e->inline_failed = CIF_MISMATCHED_ARGUMENTS;
- else
- e->inline_failed = CIF_FUNCTION_NOT_CONSIDERED;
- }
-
/* Allocate a cgraph_edge structure and fill it with data according to the
parameters of which only CALLEE can be NULL (when creating an indirect call
edge). */
--- 970,975 ----
*************** dump_cgraph_node (FILE *f, struct cgraph
*** 1899,1910 ****
ld_plugin_symbol_resolution_names[(int)node->resolution]);
if (node->local.finalized)
fprintf (f, " finalized");
- if (node->local.disregard_inline_limits)
- fprintf (f, " always_inline");
- else if (node->local.inlinable)
- fprintf (f, " inlinable");
- else if (node->local.versionable)
- fprintf (f, " versionable");
if (node->local.redefined_extern_inline)
fprintf (f, " redefined_extern_inline");
if (TREE_ASM_WRITTEN (node->decl))
--- 1877,1882 ----
Index: cgraph.h
===================================================================
*** cgraph.h (revision 172431)
--- cgraph.h (working copy)
*************** struct GTY(()) cgraph_local_info {
*** 88,107 ****
/* Set once it has been finalized so we consider it to be output. */
unsigned finalized : 1;
- /* False when there something makes inlining impossible (such as va_arg). */
- unsigned inlinable : 1;
-
- /* False when there something makes versioning impossible.
- Currently computed and used only by ipa-cp. */
- unsigned versionable : 1;
-
/* False when function calling convention and signature can not be changed.
This is the case when __builtin_apply_args is used. */
unsigned can_change_signature : 1;
- /* True when function should be inlined independently on its size. */
- unsigned disregard_inline_limits : 1;
-
/* True when the function has been originally extern inline, but it is
redefined now. */
unsigned redefined_extern_inline : 1;
--- 88,97 ----
*************** struct GTY(()) cgraph_local_info {
*** 115,135 ****
once compilation is finished. Available only with -funit-at-a-time. */
struct GTY(()) cgraph_global_info {
- /* Estimated stack frame consumption by the function. */
- HOST_WIDE_INT estimated_stack_size;
- /* Expected offset of the stack frame of inlined function. */
- HOST_WIDE_INT stack_frame_offset;
-
/* For inline clones this points to the function they will be
inlined into. */
struct cgraph_node *inlined_to;
-
- /* Estimated size of the function after inlining. */
- int time;
- int size;
-
- /* Estimated growth after inlining. INT_MIN if not computed. */
- int estimated_growth;
};
/* Information about the function that is propagated by the RTL backend.
--- 105,113 ----
Index: ipa-cp.c
===================================================================
*** ipa-cp.c (revision 172431)
--- ipa-cp.c (working copy)
*************** ipcp_versionable_function_p (struct cgra
*** 424,430 ****
/* There are a number of generic reasons functions cannot be versioned. We
also cannot remove parameters if there are type attributes such as fnspec
present. */
! if (!node->local.versionable
|| TYPE_ATTRIBUTES (TREE_TYPE (node->decl)))
return false;
--- 424,430 ----
/* There are a number of generic reasons functions cannot be versioned. We
also cannot remove parameters if there are type attributes such as fnspec
present. */
! if (!inline_summary (node)->versionable
|| TYPE_ATTRIBUTES (TREE_TYPE (node->decl)))
return false;
*************** ipcp_generate_summary (void)
*** 1577,1583 ****
/* Unreachable nodes should have been eliminated before ipcp. */
gcc_assert (node->needed || node->reachable);
! node->local.versionable = tree_versionable_function_p (node->decl);
ipa_analyze_node (node);
}
}
--- 1577,1583 ----
/* Unreachable nodes should have been eliminated before ipcp. */
gcc_assert (node->needed || node->reachable);
! inline_summary (node)->versionable = tree_versionable_function_p (node->decl);
ipa_analyze_node (node);
}
}
Index: cgraphunit.c
===================================================================
*** cgraphunit.c (revision 172431)
--- cgraphunit.c (working copy)
*************** cgraph_decide_is_function_needed (struct
*** 172,178 ****
if (flag_keep_inline_functions
&& DECL_DECLARED_INLINE_P (decl)
&& !DECL_EXTERNAL (decl)
! && !lookup_attribute ("always_inline", DECL_ATTRIBUTES (decl)))
return true;
/* If we decided it was needed before, but at the time we didn't have
--- 172,178 ----
if (flag_keep_inline_functions
&& DECL_DECLARED_INLINE_P (decl)
&& !DECL_EXTERNAL (decl)
! && !DECL_DISREGARD_INLINE_LIMITS (decl))
return true;
/* If we decided it was needed before, but at the time we didn't have
*************** cgraph_decide_is_function_needed (struct
*** 191,197 ****
to change the behavior here. */
if (((TREE_PUBLIC (decl)
|| (!optimize
! && !node->local.disregard_inline_limits
&& !DECL_DECLARED_INLINE_P (decl)
&& !(DECL_CONTEXT (decl)
&& TREE_CODE (DECL_CONTEXT (decl)) == FUNCTION_DECL)))
--- 191,197 ----
to change the behavior here. */
if (((TREE_PUBLIC (decl)
|| (!optimize
! && !DECL_DISREGARD_INLINE_LIMITS (decl)
&& !DECL_DECLARED_INLINE_P (decl)
&& !(DECL_CONTEXT (decl)
&& TREE_CODE (DECL_CONTEXT (decl)) == FUNCTION_DECL)))
*************** cgraph_analyze_function (struct cgraph_n
*** 783,793 ****
assign_assembler_name_if_neeeded (node->decl);
- /* disregard_inline_limits affects topological order of the early optimization,
- so we need to compute it ahead of rest of inline parameters. */
- node->local.disregard_inline_limits
- = DECL_DISREGARD_INLINE_LIMITS (node->decl);
-
/* Make sure to gimplify bodies only once. During analyzing a
function we lower it, which will require gimplified nested
functions, so we can end up here with an already gimplified
--- 783,788 ----
Index: lto-cgraph.c
===================================================================
*** lto-cgraph.c (revision 172431)
--- lto-cgraph.c (working copy)
*************** lto_output_node (struct lto_simple_outpu
*** 489,498 ****
bp_pack_value (&bp, node->local.local, 1);
bp_pack_value (&bp, node->local.externally_visible, 1);
bp_pack_value (&bp, node->local.finalized, 1);
- bp_pack_value (&bp, node->local.inlinable, 1);
- bp_pack_value (&bp, node->local.versionable, 1);
bp_pack_value (&bp, node->local.can_change_signature, 1);
- bp_pack_value (&bp, node->local.disregard_inline_limits, 1);
bp_pack_value (&bp, node->local.redefined_extern_inline, 1);
bp_pack_value (&bp, node->local.vtable_method, 1);
bp_pack_value (&bp, node->needed, 1);
--- 489,495 ----
*************** input_overwrite_node (struct lto_file_de
*** 928,937 ****
node->local.local = bp_unpack_value (bp, 1);
node->local.externally_visible = bp_unpack_value (bp, 1);
node->local.finalized = bp_unpack_value (bp, 1);
- node->local.inlinable = bp_unpack_value (bp, 1);
- node->local.versionable = bp_unpack_value (bp, 1);
node->local.can_change_signature = bp_unpack_value (bp, 1);
- node->local.disregard_inline_limits = bp_unpack_value (bp, 1);
node->local.redefined_extern_inline = bp_unpack_value (bp, 1);
node->local.vtable_method = bp_unpack_value (bp, 1);
node->needed = bp_unpack_value (bp, 1);
--- 925,931 ----
Index: ipa-inline.c
===================================================================
*** ipa-inline.c (revision 172431)
--- ipa-inline.c (working copy)
*************** cgraph_clone_inlined_nodes (struct cgrap
*** 156,161 ****
--- 156,162 ----
bool update_original)
{
HOST_WIDE_INT peak;
+ struct inline_summary *caller_info, *callee_info;
if (duplicate)
{
*************** cgraph_clone_inlined_nodes (struct cgrap
*** 184,190 ****
gcc_assert (!e->callee->global.inlined_to);
if (e->callee->analyzed && !DECL_EXTERNAL (e->callee->decl))
{
! overall_size -= e->callee->global.size;
nfunctions_inlined++;
}
duplicate = false;
--- 185,191 ----
gcc_assert (!e->callee->global.inlined_to);
if (e->callee->analyzed && !DECL_EXTERNAL (e->callee->decl))
{
! overall_size -= inline_summary (e->callee)->size;
nfunctions_inlined++;
}
duplicate = false;
*************** cgraph_clone_inlined_nodes (struct cgrap
*** 201,217 ****
}
}
if (e->caller->global.inlined_to)
e->callee->global.inlined_to = e->caller->global.inlined_to;
else
e->callee->global.inlined_to = e->caller;
! e->callee->global.stack_frame_offset
! = e->caller->global.stack_frame_offset
! + inline_summary (e->caller)->estimated_self_stack_size;
! peak = e->callee->global.stack_frame_offset
! + inline_summary (e->callee)->estimated_self_stack_size;
! if (e->callee->global.inlined_to->global.estimated_stack_size < peak)
! e->callee->global.inlined_to->global.estimated_stack_size = peak;
cgraph_propagate_frequency (e->callee);
/* Recursively clone all bodies. */
--- 202,221 ----
}
}
+ callee_info = inline_summary (e->callee);
+ caller_info = inline_summary (e->caller);
+
if (e->caller->global.inlined_to)
e->callee->global.inlined_to = e->caller->global.inlined_to;
else
e->callee->global.inlined_to = e->caller;
! callee_info->stack_frame_offset
! = caller_info->stack_frame_offset
! + caller_info->estimated_self_stack_size;
! peak = callee_info->stack_frame_offset
! + callee_info->estimated_self_stack_size;
! if (inline_summary (e->callee->global.inlined_to)->estimated_stack_size < peak)
! inline_summary (e->callee->global.inlined_to)->estimated_stack_size = peak;
cgraph_propagate_frequency (e->callee);
/* Recursively clone all bodies. */
*************** cgraph_mark_inline_edge (struct cgraph_e
*** 233,238 ****
--- 237,243 ----
int old_size = 0, new_size = 0;
struct cgraph_node *to = NULL;
struct cgraph_edge *curr = e;
+ struct inline_summary *info;
/* Don't inline inlined edges. */
gcc_assert (e->inline_failed);
*************** cgraph_mark_inline_edge (struct cgraph_e
*** 248,257 ****
for (;e && !e->inline_failed; e = e->caller->callers)
{
to = e->caller;
! old_size = e->caller->global.size;
new_size = estimate_size_after_inlining (to, curr);
! to->global.size = new_size;
! to->global.time = estimate_time_after_inlining (to, curr);
}
gcc_assert (curr->callee->global.inlined_to == to);
if (new_size > old_size)
--- 253,263 ----
for (;e && !e->inline_failed; e = e->caller->callers)
{
to = e->caller;
! info = inline_summary (to);
! old_size = info->size;
new_size = estimate_size_after_inlining (to, curr);
! info->size = new_size;
! info->time = estimate_time_after_inlining (to, curr);
}
gcc_assert (curr->callee->global.inlined_to == to);
if (new_size > old_size)
*************** cgraph_check_inline_limits (struct cgrap
*** 280,302 ****
int newsize;
int limit;
HOST_WIDE_INT stack_size_limit, inlined_stack;
if (to->global.inlined_to)
to = to->global.inlined_to;
/* When inlining large function body called once into small function,
take the inlined function as base for limiting the growth. */
! if (inline_summary (to)->self_size > inline_summary(what)->self_size)
! limit = inline_summary (to)->self_size;
else
! limit = inline_summary (what)->self_size;
limit += limit * PARAM_VALUE (PARAM_LARGE_FUNCTION_GROWTH) / 100;
/* Check the size after inlining against the function limits. But allow
the function to shrink if it went over the limits by forced inlining. */
newsize = estimate_size_after_inlining (to, e);
! if (newsize >= to->global.size
&& newsize > PARAM_VALUE (PARAM_LARGE_FUNCTION_INSNS)
&& newsize > limit)
{
--- 286,312 ----
int newsize;
int limit;
HOST_WIDE_INT stack_size_limit, inlined_stack;
+ struct inline_summary *info, *what_info;
if (to->global.inlined_to)
to = to->global.inlined_to;
+ info = inline_summary (to);
+ what_info = inline_summary (what);
+
/* When inlining large function body called once into small function,
take the inlined function as base for limiting the growth. */
! if (info->self_size > what_info->self_size)
! limit = info->self_size;
else
! limit = what_info->self_size;
limit += limit * PARAM_VALUE (PARAM_LARGE_FUNCTION_GROWTH) / 100;
/* Check the size after inlining against the function limits. But allow
the function to shrink if it went over the limits by forced inlining. */
newsize = estimate_size_after_inlining (to, e);
! if (newsize >= info->size
&& newsize > PARAM_VALUE (PARAM_LARGE_FUNCTION_INSNS)
&& newsize > limit)
{
*************** cgraph_check_inline_limits (struct cgrap
*** 305,317 ****
return false;
}
! stack_size_limit = inline_summary (to)->estimated_self_stack_size;
stack_size_limit += stack_size_limit * PARAM_VALUE (PARAM_STACK_FRAME_GROWTH) / 100;
! inlined_stack = (to->global.stack_frame_offset
! + inline_summary (to)->estimated_self_stack_size
! + what->global.estimated_stack_size);
if (inlined_stack > stack_size_limit
&& inlined_stack > PARAM_VALUE (PARAM_LARGE_STACK_FRAME))
{
--- 315,327 ----
return false;
}
! stack_size_limit = info->estimated_self_stack_size;
stack_size_limit += stack_size_limit * PARAM_VALUE (PARAM_STACK_FRAME_GROWTH) / 100;
! inlined_stack = (info->stack_frame_offset
! + info->estimated_self_stack_size
! + what_info->estimated_stack_size);
if (inlined_stack > stack_size_limit
&& inlined_stack > PARAM_VALUE (PARAM_LARGE_STACK_FRAME))
{
*************** static bool
*** 328,335 ****
cgraph_default_inline_p (struct cgraph_node *n, cgraph_inline_failed_t *reason)
{
tree decl = n->decl;
! if (n->local.disregard_inline_limits)
return true;
if (!flag_inline_small_functions && !DECL_DECLARED_INLINE_P (decl))
--- 338,346 ----
cgraph_default_inline_p (struct cgraph_node *n, cgraph_inline_failed_t *reason)
{
tree decl = n->decl;
+ struct inline_summary *info = inline_summary (n);
! if (info->disregard_inline_limits)
return true;
if (!flag_inline_small_functions && !DECL_DECLARED_INLINE_P (decl))
*************** cgraph_default_inline_p (struct cgraph_n
*** 354,360 ****
if (DECL_DECLARED_INLINE_P (decl))
{
! if (n->global.size >= MAX_INLINE_INSNS_SINGLE)
{
if (reason)
*reason = CIF_MAX_INLINE_INSNS_SINGLE_LIMIT;
--- 365,371 ----
if (DECL_DECLARED_INLINE_P (decl))
{
! if (info->size >= MAX_INLINE_INSNS_SINGLE)
{
if (reason)
*reason = CIF_MAX_INLINE_INSNS_SINGLE_LIMIT;
*************** cgraph_default_inline_p (struct cgraph_n
*** 363,369 ****
}
else
{
! if (n->global.size >= MAX_INLINE_INSNS_AUTO)
{
if (reason)
*reason = CIF_MAX_INLINE_INSNS_AUTO_LIMIT;
--- 374,380 ----
}
else
{
! if (info->size >= MAX_INLINE_INSNS_AUTO)
{
if (reason)
*reason = CIF_MAX_INLINE_INSNS_AUTO_LIMIT;
*************** cgraph_edge_badness (struct cgraph_edge
*** 385,392 ****
{
gcov_type badness;
int growth;
! if (edge->callee->local.disregard_inline_limits)
return INT_MIN;
growth = estimate_edge_growth (edge);
--- 396,404 ----
{
gcov_type badness;
int growth;
+ struct inline_summary *callee_info = inline_summary (edge->callee);
! if (callee_info->disregard_inline_limits)
return INT_MIN;
growth = estimate_edge_growth (edge);
*************** cgraph_edge_badness (struct cgraph_edge
*** 398,408 ****
cgraph_node_name (edge->callee));
fprintf (dump_file, " growth %i, time %i-%i, size %i-%i\n",
growth,
! edge->callee->global.time,
! inline_summary (edge->callee)->time_inlining_benefit
+ edge->call_stmt_time,
! edge->callee->global.size,
! inline_summary (edge->callee)->size_inlining_benefit
+ edge->call_stmt_size);
}
--- 410,420 ----
cgraph_node_name (edge->callee));
fprintf (dump_file, " growth %i, time %i-%i, size %i-%i\n",
growth,
! callee_info->time,
! callee_info->time_inlining_benefit
+ edge->call_stmt_time,
! callee_info->size,
! callee_info->size_inlining_benefit
+ edge->call_stmt_size);
}
*************** cgraph_edge_badness (struct cgraph_edge
*** 422,428 ****
badness =
((int)
((double) edge->count * INT_MIN / max_count / (max_benefit + 1)) *
! (inline_summary (edge->callee)->time_inlining_benefit
+ edge->call_stmt_time + 1)) / growth;
if (dump)
{
--- 434,440 ----
badness =
((int)
((double) edge->count * INT_MIN / max_count / (max_benefit + 1)) *
! (callee_info->time_inlining_benefit
+ edge->call_stmt_time + 1)) / growth;
if (dump)
{
*************** cgraph_edge_badness (struct cgraph_edge
*** 453,461 ****
int growth_for_all;
badness = growth * 10000;
benefitperc =
! 100 * (inline_summary (edge->callee)->time_inlining_benefit
+ edge->call_stmt_time)
! / (edge->callee->global.time + 1) + 1;
benefitperc = MIN (benefitperc, 100);
div *= benefitperc;
--- 465,473 ----
int growth_for_all;
badness = growth * 10000;
benefitperc =
! 100 * (callee_info->time_inlining_benefit
+ edge->call_stmt_time)
! / (callee_info->time + 1) + 1;
benefitperc = MIN (benefitperc, 100);
div *= benefitperc;
*************** update_caller_keys (fibheap_t heap, stru
*** 543,555 ****
struct cgraph_edge *edge;
cgraph_inline_failed_t failed_reason;
! if (!node->local.inlinable
|| cgraph_function_body_availability (node) <= AVAIL_OVERWRITABLE
|| node->global.inlined_to)
return;
if (!bitmap_set_bit (updated_nodes, node->uid))
return;
! node->global.estimated_growth = INT_MIN;
/* See if there is something to do. */
for (edge = node->callers; edge; edge = edge->next_caller)
--- 555,567 ----
struct cgraph_edge *edge;
cgraph_inline_failed_t failed_reason;
! if (!inline_summary (node)->inlinable
|| cgraph_function_body_availability (node) <= AVAIL_OVERWRITABLE
|| node->global.inlined_to)
return;
if (!bitmap_set_bit (updated_nodes, node->uid))
return;
! inline_summary (node)->estimated_growth = INT_MIN;
/* See if there is something to do. */
for (edge = node->callers; edge; edge = edge->next_caller)
*************** update_callee_keys (fibheap_t heap, stru
*** 586,592 ****
bitmap updated_nodes)
{
struct cgraph_edge *e = node->callees;
! node->global.estimated_growth = INT_MIN;
if (!e)
return;
--- 598,604 ----
bitmap updated_nodes)
{
struct cgraph_edge *e = node->callees;
! inline_summary (node)->estimated_growth = INT_MIN;
if (!e)
return;
*************** update_callee_keys (fibheap_t heap, stru
*** 596,606 ****
else
{
if (e->inline_failed
! && e->callee->local.inlinable
&& cgraph_function_body_availability (e->callee) >= AVAIL_AVAILABLE
&& !bitmap_bit_p (updated_nodes, e->callee->uid))
{
! node->global.estimated_growth = INT_MIN;
/* If function becomes uninlinable, we need to remove it from the heap. */
if (!cgraph_default_inline_p (e->callee, &e->inline_failed))
update_caller_keys (heap, e->callee, updated_nodes);
--- 608,618 ----
else
{
if (e->inline_failed
! && inline_summary (e->callee)->inlinable
&& cgraph_function_body_availability (e->callee) >= AVAIL_AVAILABLE
&& !bitmap_bit_p (updated_nodes, e->callee->uid))
{
! inline_summary (node)->estimated_growth = INT_MIN;
/* If function becomes uninlinable, we need to remove it from the heap. */
if (!cgraph_default_inline_p (e->callee, &e->inline_failed))
update_caller_keys (heap, e->callee, updated_nodes);
*************** update_all_callee_keys (fibheap_t heap,
*** 632,638 ****
bitmap updated_nodes)
{
struct cgraph_edge *e = node->callees;
! node->global.estimated_growth = INT_MIN;
if (!e)
return;
--- 644,650 ----
bitmap updated_nodes)
{
struct cgraph_edge *e = node->callees;
! inline_summary (node)->estimated_growth = INT_MIN;
if (!e)
return;
*************** cgraph_decide_recursive_inlining (struct
*** 709,715 ****
/* It does not make sense to recursively inline always-inline functions
as we are going to sorry() on the remaining calls anyway. */
! if (node->local.disregard_inline_limits
&& lookup_attribute ("always_inline", DECL_ATTRIBUTES (node->decl)))
return false;
--- 721,727 ----
/* It does not make sense to recursively inline always-inline functions
as we are going to sorry() on the remaining calls anyway. */
! if (inline_summary (node)->disregard_inline_limits
&& lookup_attribute ("always_inline", DECL_ATTRIBUTES (node->decl)))
return false;
*************** cgraph_decide_recursive_inlining (struct
*** 811,818 ****
if (dump_file)
fprintf (dump_file,
"\n Inlined %i times, body grown from size %i to %i, time %i to %i\n", n,
! master_clone->global.size, node->global.size,
! master_clone->global.time, node->global.time);
/* Remove master clone we used for inlining. We rely that clones inlined
into master clone gets queued just before master clone so we don't
--- 823,830 ----
if (dump_file)
fprintf (dump_file,
"\n Inlined %i times, body grown from size %i to %i, time %i to %i\n", n,
! inline_summary (master_clone)->size, inline_summary (node)->size,
! inline_summary (master_clone)->time, inline_summary (node)->time);
/* Remove master clone we used for inlining. We rely that clones inlined
into master clone gets queued just before master clone so we don't
*************** add_new_edges_to_heap (fibheap_t heap, V
*** 870,876 ****
struct cgraph_edge *edge = VEC_pop (cgraph_edge_p, new_edges);
gcc_assert (!edge->aux);
! if (edge->callee->local.inlinable
&& edge->inline_failed
&& cgraph_default_inline_p (edge->callee, &edge->inline_failed))
edge->aux = fibheap_insert (heap, cgraph_edge_badness (edge, false), edge);
--- 882,888 ----
struct cgraph_edge *edge = VEC_pop (cgraph_edge_p, new_edges);
gcc_assert (!edge->aux);
! if (inline_summary (edge->callee)->inlinable
&& edge->inline_failed
&& cgraph_default_inline_p (edge->callee, &edge->inline_failed))
edge->aux = fibheap_insert (heap, cgraph_edge_badness (edge, false), edge);
*************** cgraph_decide_inlining_of_small_function
*** 905,930 ****
/* Put all inline candidates into the heap. */
for (node = cgraph_nodes; node; node = node->next)
! {
! if (!node->local.inlinable || !node->callers)
! continue;
! if (dump_file)
! fprintf (dump_file, "Considering inline candidate %s.\n", cgraph_node_name (node));
! node->global.estimated_growth = INT_MIN;
! if (!cgraph_default_inline_p (node, &failed_reason))
! {
! cgraph_set_inline_failed (node, failed_reason);
! continue;
! }
! for (edge = node->callers; edge; edge = edge->next_caller)
! if (edge->inline_failed)
{
! gcc_assert (!edge->aux);
! edge->aux = fibheap_insert (heap, cgraph_edge_badness (edge, false), edge);
}
! }
max_size = compute_max_insns (overall_size);
min_size = overall_size;
--- 917,953 ----
/* Put all inline candidates into the heap. */
for (node = cgraph_nodes; node; node = node->next)
! if (node->analyzed)
! {
! struct inline_summary *info = inline_summary (node);
! if (!info->inlinable || !node->callers)
! {
! struct cgraph_edge *e;
! for (e = node->callers; e; e = e->next_caller)
! {
! gcc_assert (e->inline_failed);
! e->inline_failed = CIF_FUNCTION_NOT_INLINABLE;
! }
! continue;
! }
! if (dump_file)
! fprintf (dump_file, "Considering inline candidate %s.\n", cgraph_node_name (node));
! info->estimated_growth = INT_MIN;
! if (!cgraph_default_inline_p (node, &failed_reason))
{
! cgraph_set_inline_failed (node, failed_reason);
! continue;
}
!
! for (edge = node->callers; edge; edge = edge->next_caller)
! if (edge->inline_failed)
! {
! gcc_assert (!edge->aux);
! edge->aux = fibheap_insert (heap, cgraph_edge_badness (edge, false), edge);
! }
! }
max_size = compute_max_insns (overall_size);
min_size = overall_size;
*************** cgraph_decide_inlining_of_small_function
*** 963,969 ****
fprintf (dump_file,
"\nConsidering %s with %i size\n",
cgraph_node_name (edge->callee),
! edge->callee->global.size);
fprintf (dump_file,
" to be inlined into %s in %s:%i\n"
" Estimated growth after inlined into all callees is %+i insns.\n"
--- 986,992 ----
fprintf (dump_file,
"\nConsidering %s with %i size\n",
cgraph_node_name (edge->callee),
! inline_summary (edge->callee)->size);
fprintf (dump_file,
" to be inlined into %s in %s:%i\n"
" Estimated growth after inlined into all callees is %+i insns.\n"
*************** cgraph_decide_inlining_of_small_function
*** 1007,1013 ****
if (where->global.inlined_to)
{
edge->inline_failed
! = (edge->callee->local.disregard_inline_limits
? CIF_RECURSIVE_INLINING : CIF_UNSPECIFIED);
if (dump_file)
fprintf (dump_file, " inline_failed:Recursive inlining performed only for function itself.\n");
--- 1030,1036 ----
if (where->global.inlined_to)
{
edge->inline_failed
! = (inline_summary (edge->callee)->disregard_inline_limits
? CIF_RECURSIVE_INLINING : CIF_UNSPECIFIED);
if (dump_file)
fprintf (dump_file, " inline_failed:Recursive inlining performed only for function itself.\n");
*************** cgraph_decide_inlining_of_small_function
*** 1015,1021 ****
}
}
! if (edge->callee->local.disregard_inline_limits)
;
else if (!cgraph_maybe_hot_edge_p (edge))
not_good = CIF_UNLIKELY_CALL;
--- 1038,1044 ----
}
}
! if (inline_summary (edge->callee)->disregard_inline_limits)
;
else if (!cgraph_maybe_hot_edge_p (edge))
not_good = CIF_UNLIKELY_CALL;
*************** cgraph_decide_inlining_of_small_function
*** 1112,1119 ****
" Inlined into %s which now has time %i and size %i,"
"net change of %+i.\n",
cgraph_node_name (edge->caller),
! edge->caller->global.time,
! edge->caller->global.size,
overall_size - old_size);
}
if (min_size > overall_size)
--- 1135,1142 ----
" Inlined into %s which now has time %i and size %i,"
"net change of %+i.\n",
cgraph_node_name (edge->caller),
! inline_summary (edge->caller)->time,
! inline_summary (edge->caller)->size,
overall_size - old_size);
}
if (min_size > overall_size)
*************** cgraph_decide_inlining_of_small_function
*** 1142,1148 ****
fprintf (dump_file,
"\nSkipping %s with %i size\n",
cgraph_node_name (edge->callee),
! edge->callee->global.size);
fprintf (dump_file,
" called by %s in %s:%i\n"
" Estimated growth after inlined into all callees is %+i insns.\n"
--- 1165,1171 ----
fprintf (dump_file,
"\nSkipping %s with %i size\n",
cgraph_node_name (edge->callee),
! inline_summary (edge->callee)->size);
fprintf (dump_file,
" called by %s in %s:%i\n"
" Estimated growth after inlined into all callees is %+i insns.\n"
*************** cgraph_decide_inlining_of_small_function
*** 1159,1165 ****
if (dump_flags & TDF_DETAILS)
cgraph_edge_badness (edge, true);
}
! if (!edge->callee->local.disregard_inline_limits && edge->inline_failed)
edge->inline_failed = CIF_INLINE_UNIT_GROWTH_LIMIT;
}
--- 1182,1188 ----
if (dump_flags & TDF_DETAILS)
cgraph_edge_badness (edge, true);
}
! if (!inline_summary (edge->callee)->disregard_inline_limits && edge->inline_failed)
edge->inline_failed = CIF_INLINE_UNIT_GROWTH_LIMIT;
}
*************** cgraph_decide_inlining (void)
*** 1287,1299 ****
if (node->analyzed)
{
struct cgraph_edge *e;
! gcc_assert (inline_summary (node)->self_size == node->global.size);
if (!DECL_EXTERNAL (node->decl))
! initial_size += node->global.size;
for (e = node->callees; e; e = e->next_callee)
{
! int benefit = (inline_summary (node)->time_inlining_benefit
+ e->call_stmt_time);
if (max_count < e->count)
max_count = e->count;
--- 1310,1323 ----
if (node->analyzed)
{
struct cgraph_edge *e;
+ struct inline_summary *info = inline_summary (node);
! gcc_assert (info->self_size == info->size);
if (!DECL_EXTERNAL (node->decl))
! initial_size += info->size;
for (e = node->callees; e; e = e->next_callee)
{
! int benefit = (info->time_inlining_benefit
+ e->call_stmt_time);
if (max_count < e->count)
max_count = e->count;
*************** cgraph_decide_inlining (void)
*** 1362,1368 ****
&& !node->callers->next_caller
&& !node->global.inlined_to
&& cgraph_will_be_removed_from_program_if_no_direct_calls (node)
! && node->local.inlinable
&& cgraph_function_body_availability (node) >= AVAIL_AVAILABLE
&& node->callers->inline_failed
&& node->callers->caller != node
--- 1386,1392 ----
&& !node->callers->next_caller
&& !node->global.inlined_to
&& cgraph_will_be_removed_from_program_if_no_direct_calls (node)
! && inline_summary (node)->inlinable
&& cgraph_function_body_availability (node) >= AVAIL_AVAILABLE
&& node->callers->inline_failed
&& node->callers->caller != node
*************** cgraph_decide_inlining (void)
*** 1377,1387 ****
{
fprintf (dump_file,
"\nConsidering %s size %i.\n",
! cgraph_node_name (node), node->global.size);
fprintf (dump_file,
" Called once from %s %i insns.\n",
cgraph_node_name (node->callers->caller),
! node->callers->caller->global.size);
}
if (cgraph_check_inline_limits (node->callers, &reason))
--- 1401,1411 ----
{
fprintf (dump_file,
"\nConsidering %s size %i.\n",
! cgraph_node_name (node), inline_summary (node)->size);
fprintf (dump_file,
" Called once from %s %i insns.\n",
cgraph_node_name (node->callers->caller),
! inline_summary (node->callers->caller)->size);
}
if (cgraph_check_inline_limits (node->callers, &reason))
*************** cgraph_decide_inlining (void)
*** 1393,1399 ****
" Inlined into %s which now has %i size"
" for a net change of %+i size.\n",
cgraph_node_name (caller),
! caller->global.size,
overall_size - old_size);
}
else
--- 1417,1423 ----
" Inlined into %s which now has %i size"
" for a net change of %+i size.\n",
cgraph_node_name (caller),
! inline_summary (caller)->size,
overall_size - old_size);
}
else
*************** leaf_node_p (struct cgraph_node *n)
*** 1442,1448 ****
static bool
cgraph_edge_early_inlinable_p (struct cgraph_edge *e, FILE *file)
{
! if (!e->callee->local.inlinable)
{
if (file)
fprintf (file, "Not inlining: Function not inlinable.\n");
--- 1466,1472 ----
static bool
cgraph_edge_early_inlinable_p (struct cgraph_edge *e, FILE *file)
{
! if (!inline_summary (e->callee)->inlinable)
{
if (file)
fprintf (file, "Not inlining: Function not inlinable.\n");
*************** cgraph_perform_always_inlining (struct c
*** 1482,1488 ****
for (e = node->callees; e; e = e->next_callee)
{
! if (!e->callee->local.disregard_inline_limits)
continue;
if (dump_file)
--- 1506,1512 ----
for (e = node->callees; e; e = e->next_callee)
{
! if (!inline_summary (e->callee)->disregard_inline_limits)
continue;
if (dump_file)
*************** cgraph_decide_inlining_incrementally (st
*** 1524,1539 ****
/* Never inline regular functions into always-inline functions
during incremental inlining. */
! if (node->local.disregard_inline_limits)
return false;
for (e = node->callees; e; e = e->next_callee)
{
int allowed_growth = 0;
! if (!e->callee->local.inlinable
|| !e->inline_failed
! || e->callee->local.disregard_inline_limits)
continue;
/* Do not consider functions not declared inline. */
--- 1548,1563 ----
/* Never inline regular functions into always-inline functions
during incremental inlining. */
! if (inline_summary (node)->disregard_inline_limits)
return false;
for (e = node->callees; e; e = e->next_callee)
{
int allowed_growth = 0;
! if (!inline_summary (e->callee)->inlinable
|| !e->inline_failed
! || inline_summary (e->callee)->disregard_inline_limits)
continue;
/* Do not consider functions not declared inline. */
Index: ipa-inline.h
===================================================================
*** ipa-inline.h (revision 172431)
--- ipa-inline.h (working copy)
*************** along with GCC; see the file COPYING3.
*** 23,31 ****
struct inline_summary
{
/* Estimated stack frame consumption by the function. */
HOST_WIDE_INT estimated_self_stack_size;
-
/* Size of the function body. */
int self_size;
/* How many instructions are likely going to disappear after inlining. */
--- 23,32 ----
struct inline_summary
{
+ /* Information about the function body itself. */
+
/* Estimated stack frame consumption by the function. */
HOST_WIDE_INT estimated_self_stack_size;
/* Size of the function body. */
int self_size;
/* How many instructions are likely going to disappear after inlining. */
*************** struct inline_summary
*** 34,39 ****
--- 35,63 ----
int self_time;
/* How much time is going to be saved by inlining. */
int time_inlining_benefit;
+
+ /* False when there something makes inlining impossible (such as va_arg). */
+ unsigned inlinable : 1;
+ /* False when there something makes versioning impossible.
+ Currently computed and used only by ipa-cp. */
+ unsigned versionable : 1;
+ /* True when function should be inlined independently on its size. */
+ unsigned disregard_inline_limits : 1;
+
+ /* Information about function that will result after applying all the
+ inline decisions present in the callgraph. Generally kept up to
+ date only for functions that are not inline clones. */
+
+ /* Estimated stack frame consumption by the function. */
+ HOST_WIDE_INT estimated_stack_size;
+ /* Expected offset of the stack frame of inlined function. */
+ HOST_WIDE_INT stack_frame_offset;
+ /* Estimated size of the function after inlining. */
+ int time;
+ int size;
+ /* Cached estimated growth after inlining.
+ INT_MIN if not computed. */
+ int estimated_growth;
};
typedef struct inline_summary inline_summary_t;
*************** void inline_generate_summary (void);
*** 47,52 ****
--- 71,77 ----
void inline_read_summary (void);
void inline_write_summary (cgraph_node_set, varpool_node_set);
void inline_free_summary (void);
+ void initialize_inline_failed (struct cgraph_edge *);
int estimate_time_after_inlining (struct cgraph_node *, struct cgraph_edge *);
int estimate_size_after_inlining (struct cgraph_node *, struct cgraph_edge *);
int estimate_growth (struct cgraph_node *);
*************** static inline int
*** 63,72 ****
estimate_edge_growth (struct cgraph_edge *edge)
{
int call_stmt_size;
call_stmt_size = edge->call_stmt_size;
gcc_checking_assert (call_stmt_size);
! return (edge->callee->global.size
! - inline_summary (edge->callee)->size_inlining_benefit
- call_stmt_size);
}
-
--- 88,97 ----
estimate_edge_growth (struct cgraph_edge *edge)
{
int call_stmt_size;
+ struct inline_summary *info = inline_summary (edge->callee);
call_stmt_size = edge->call_stmt_size;
gcc_checking_assert (call_stmt_size);
! return (info->size
! - info->size_inlining_benefit
- call_stmt_size);
}
Index: ipa-split.c
===================================================================
*** ipa-split.c (revision 172431)
--- ipa-split.c (working copy)
*************** along with GCC; see the file COPYING3.
*** 92,97 ****
--- 92,98 ----
#include "fibheap.h"
#include "params.h"
#include "gimple-pretty-print.h"
+ #include "ipa-inline.h"
/* Per basic block info. */
*************** execute_split_functions (void)
*** 1281,1293 ****
}
/* This can be relaxed; function might become inlinable after splitting
away the uninlinable part. */
! if (!node->local.inlinable)
{
if (dump_file)
fprintf (dump_file, "Not splitting: not inlinable.\n");
return 0;
}
! if (node->local.disregard_inline_limits)
{
if (dump_file)
fprintf (dump_file, "Not splitting: disregarding inline limits.\n");
--- 1282,1294 ----
}
/* This can be relaxed; function might become inlinable after splitting
away the uninlinable part. */
! if (!inline_summary (node)->inlinable)
{
if (dump_file)
fprintf (dump_file, "Not splitting: not inlinable.\n");
return 0;
}
! if (DECL_DISREGARD_INLINE_LIMITS (node->decl))
{
if (dump_file)
fprintf (dump_file, "Not splitting: disregarding inline limits.\n");
Index: ipa.c
===================================================================
*** ipa.c (revision 172431)
--- ipa.c (working copy)
*************** cgraph_postorder (struct cgraph_node **o
*** 78,85 ****
/* Break possible cycles involving always-inline
functions by ignoring edges from always-inline
functions to non-always-inline functions. */
! if (edge->caller->local.disregard_inline_limits
! && !edge->callee->local.disregard_inline_limits)
continue;
if (!edge->caller->aux)
{
--- 78,85 ----
/* Break possible cycles involving always-inline
functions by ignoring edges from always-inline
functions to non-always-inline functions. */
! if (DECL_DISREGARD_INLINE_LIMITS (edge->caller->decl)
! && !DECL_DISREGARD_INLINE_LIMITS (edge->callee->decl))
continue;
if (!edge->caller->aux)
{
*************** cgraph_remove_unreachable_nodes (bool be
*** 380,386 ****
cgraph_node_remove_callees (node);
ipa_remove_all_references (&node->ref_list);
node->analyzed = false;
- node->local.inlinable = false;
}
if (!node->aux)
{
--- 380,385 ----
*************** cgraph_remove_unreachable_nodes (bool be
*** 421,427 ****
if (!clone)
{
cgraph_release_function_body (node);
- node->local.inlinable = false;
if (node->prev_sibling_clone)
node->prev_sibling_clone->next_sibling_clone = node->next_sibling_clone;
else if (node->clone_of)
--- 420,425 ----
*************** record_cdtor_fn (struct cgraph_node *nod
*** 1629,1635 ****
if (DECL_STATIC_DESTRUCTOR (node->decl))
VEC_safe_push (tree, heap, static_dtors, node->decl);
node = cgraph_get_node (node->decl);
! node->local.disregard_inline_limits = 1;
}
/* Define global constructors/destructor functions for the CDTORS, of
--- 1627,1633 ----
if (DECL_STATIC_DESTRUCTOR (node->decl))
VEC_safe_push (tree, heap, static_dtors, node->decl);
node = cgraph_get_node (node->decl);
! DECL_DISREGARD_INLINE_LIMITS (node->decl) = 1;
}
/* Define global constructors/destructor functions for the CDTORS, of
Index: ipa-inline-analysis.c
===================================================================
*** ipa-inline-analysis.c (revision 172431)
--- ipa-inline-analysis.c (working copy)
*************** inline_summary_alloc (void)
*** 100,110 ****
static void
inline_node_removal_hook (struct cgraph_node *node, void *data ATTRIBUTE_UNUSED)
{
if (VEC_length (inline_summary_t, inline_summary_vec)
<= (unsigned)node->uid)
return;
! memset (inline_summary (node),
! 0, sizeof (inline_summary_t));
}
/* Hook that is called by cgraph.c when a node is duplicated. */
--- 100,112 ----
static void
inline_node_removal_hook (struct cgraph_node *node, void *data ATTRIBUTE_UNUSED)
{
+ struct inline_summary *info;
if (VEC_length (inline_summary_t, inline_summary_vec)
<= (unsigned)node->uid)
return;
! info = inline_summary (node);
! info->estimated_growth = INT_MIN;
! memset (info, 0, sizeof (inline_summary_t));
}
/* Hook that is called by cgraph.c when a node is duplicated. */
*************** static void
*** 113,121 ****
inline_node_duplication_hook (struct cgraph_node *src, struct cgraph_node *dst,
ATTRIBUTE_UNUSED void *data)
{
inline_summary_alloc ();
! memcpy (inline_summary (dst), inline_summary (src),
sizeof (struct inline_summary));
}
static void
--- 115,126 ----
inline_node_duplication_hook (struct cgraph_node *src, struct cgraph_node *dst,
ATTRIBUTE_UNUSED void *data)
{
+ struct inline_summary *info;
inline_summary_alloc ();
! info = inline_summary (dst);
! memcpy (info, inline_summary (src),
sizeof (struct inline_summary));
+ info->estimated_growth = INT_MIN;
}
static void
*************** dump_inline_summary (FILE *f, struct cgr
*** 124,141 ****
if (node->analyzed)
{
struct inline_summary *s = inline_summary (node);
! fprintf (f, "Inline summary for %s/%i\n", cgraph_node_name (node),
node->uid);
! fprintf (f, " self time: %i, benefit: %i\n",
s->self_time, s->time_inlining_benefit);
! fprintf (f, " global time: %i\n", node->global.time);
fprintf (f, " self size: %i, benefit: %i\n",
s->self_size, s->size_inlining_benefit);
! fprintf (f, " global size: %i", node->global.size);
fprintf (f, " self stack: %i\n",
(int)s->estimated_self_stack_size);
fprintf (f, " global stack: %i\n\n",
! (int)node->global.estimated_stack_size);
}
}
--- 129,152 ----
if (node->analyzed)
{
struct inline_summary *s = inline_summary (node);
! fprintf (f, "Inline summary for %s/%i", cgraph_node_name (node),
node->uid);
! if (s->disregard_inline_limits)
! fprintf (f, " always_inline");
! if (s->inlinable)
! fprintf (f, " inlinable");
! if (s->versionable)
! fprintf (f, " versionable");
! fprintf (f, "\n self time: %i, benefit: %i\n",
s->self_time, s->time_inlining_benefit);
! fprintf (f, " global time: %i\n", s->time);
fprintf (f, " self size: %i, benefit: %i\n",
s->self_size, s->size_inlining_benefit);
! fprintf (f, " global size: %i", s->size);
fprintf (f, " self stack: %i\n",
(int)s->estimated_self_stack_size);
fprintf (f, " global stack: %i\n\n",
! (int)s->estimated_stack_size);
}
}
*************** dump_inline_summaries (FILE *f)
*** 155,160 ****
--- 166,191 ----
dump_inline_summary (f, node);
}
+ /* Give initial reasons why inlining would fail on EDGE. This gets either
+ nullified or usually overwritten by more precise reasons later. */
+
+ void
+ initialize_inline_failed (struct cgraph_edge *e)
+ {
+ struct cgraph_node *callee = e->callee;
+
+ if (e->indirect_unknown_callee)
+ e->inline_failed = CIF_INDIRECT_UNKNOWN_CALL;
+ else if (!callee->analyzed)
+ e->inline_failed = CIF_BODY_NOT_AVAILABLE;
+ else if (callee->local.redefined_extern_inline)
+ e->inline_failed = CIF_REDEFINED_EXTERN_INLINE;
+ else if (e->call_stmt && gimple_call_cannot_inline_p (e->call_stmt))
+ e->inline_failed = CIF_MISMATCHED_ARGUMENTS;
+ else
+ e->inline_failed = CIF_FUNCTION_NOT_CONSIDERED;
+ }
+
/* See if statement might disappear after inlining.
0 - means not eliminated
1 - half of statements goes away
*************** compute_inline_parameters (struct cgraph
*** 317,340 ****
{
HOST_WIDE_INT self_stack_size;
struct cgraph_edge *e;
gcc_assert (!node->global.inlined_to);
inline_summary_alloc ();
/* Estimate the stack size for the function if we're optimizing. */
self_stack_size = optimize ? estimated_stack_frame_size (node) : 0;
! inline_summary (node)->estimated_self_stack_size = self_stack_size;
! node->global.estimated_stack_size = self_stack_size;
! node->global.stack_frame_offset = 0;
/* Can this function be inlined at all? */
! node->local.inlinable = tree_inlinable_function_p (node->decl);
! if (!node->local.inlinable)
! node->local.disregard_inline_limits = 0;
/* Inlinable functions always can change signature. */
! if (node->local.inlinable)
node->local.can_change_signature = true;
else
{
--- 348,374 ----
{
HOST_WIDE_INT self_stack_size;
struct cgraph_edge *e;
+ struct inline_summary *info;
gcc_assert (!node->global.inlined_to);
inline_summary_alloc ();
+ info = inline_summary (node);
+
/* Estimate the stack size for the function if we're optimizing. */
self_stack_size = optimize ? estimated_stack_frame_size (node) : 0;
! info->estimated_self_stack_size = self_stack_size;
! info->estimated_stack_size = self_stack_size;
! info->stack_frame_offset = 0;
/* Can this function be inlined at all? */
! info->inlinable = tree_inlinable_function_p (node->decl);
! if (!info->inlinable)
! info->disregard_inline_limits = 0;
/* Inlinable functions always can change signature. */
! if (info->inlinable)
node->local.can_change_signature = true;
else
{
*************** compute_inline_parameters (struct cgraph
*** 349,356 ****
estimate_function_body_sizes (node);
/* Inlining characteristics are maintained by the cgraph_mark_inline. */
! node->global.time = inline_summary (node)->self_time;
! node->global.size = inline_summary (node)->self_size;
}
--- 383,395 ----
estimate_function_body_sizes (node);
/* Inlining characteristics are maintained by the cgraph_mark_inline. */
! info->time = info->self_time;
! info->size = info->self_size;
! info->estimated_growth = INT_MIN;
! info->stack_frame_offset = 0;
! info->estimated_stack_size = info->estimated_self_stack_size;
! info->disregard_inline_limits
! = DECL_DISREGARD_INLINE_LIMITS (node->decl);
}
*************** static inline int
*** 390,399 ****
estimate_edge_time (struct cgraph_edge *edge)
{
int call_stmt_time;
call_stmt_time = edge->call_stmt_time;
gcc_checking_assert (call_stmt_time);
! return (((gcov_type)edge->callee->global.time
! - inline_summary (edge->callee)->time_inlining_benefit
- call_stmt_time) * edge->frequency
+ CGRAPH_FREQ_BASE / 2) / CGRAPH_FREQ_BASE;
}
--- 429,440 ----
estimate_edge_time (struct cgraph_edge *edge)
{
int call_stmt_time;
+ struct inline_summary *info = inline_summary (edge->callee);
+
call_stmt_time = edge->call_stmt_time;
gcc_checking_assert (call_stmt_time);
! return (((gcov_type)info->time
! - info->time_inlining_benefit
- call_stmt_time) * edge->frequency
+ CGRAPH_FREQ_BASE / 2) / CGRAPH_FREQ_BASE;
}
*************** int
*** 405,411 ****
estimate_time_after_inlining (struct cgraph_node *node,
struct cgraph_edge *edge)
{
! gcov_type time = node->global.time + estimate_edge_time (edge);
if (time < 0)
time = 0;
if (time > MAX_TIME)
--- 446,452 ----
estimate_time_after_inlining (struct cgraph_node *node,
struct cgraph_edge *edge)
{
! gcov_type time = inline_summary (node)->time + estimate_edge_time (edge);
if (time < 0)
time = 0;
if (time > MAX_TIME)
*************** int
*** 421,427 ****
estimate_size_after_inlining (struct cgraph_node *node,
struct cgraph_edge *edge)
{
! int size = node->global.size + estimate_edge_growth (edge);
gcc_assert (size >= 0);
return size;
}
--- 462,468 ----
estimate_size_after_inlining (struct cgraph_node *node,
struct cgraph_edge *edge)
{
! int size = inline_summary (node)->size + estimate_edge_growth (edge);
gcc_assert (size >= 0);
return size;
}
*************** estimate_growth (struct cgraph_node *nod
*** 435,443 ****
int growth = 0;
struct cgraph_edge *e;
bool self_recursive = false;
! if (node->global.estimated_growth != INT_MIN)
! return node->global.estimated_growth;
for (e = node->callers; e; e = e->next_caller)
{
--- 476,485 ----
int growth = 0;
struct cgraph_edge *e;
bool self_recursive = false;
+ struct inline_summary *info = inline_summary (node);
! if (info->estimated_growth != INT_MIN)
! return info->estimated_growth;
for (e = node->callers; e; e = e->next_caller)
{
*************** estimate_growth (struct cgraph_node *nod
*** 453,467 ****
some inlining. */
if (cgraph_will_be_removed_from_program_if_no_direct_calls (node)
&& !DECL_EXTERNAL (node->decl) && !self_recursive)
! growth -= node->global.size;
/* COMDAT functions are very often not shared across multiple units since they
come from various template instantiations. Take this into account. */
else if (DECL_COMDAT (node->decl) && !self_recursive
&& cgraph_can_remove_if_no_direct_calls_p (node))
! growth -= (node->global.size
* (100 - PARAM_VALUE (PARAM_COMDAT_SHARING_PROBABILITY)) + 50) / 100;
! node->global.estimated_growth = growth;
return growth;
}
--- 495,509 ----
some inlining. */
if (cgraph_will_be_removed_from_program_if_no_direct_calls (node)
&& !DECL_EXTERNAL (node->decl) && !self_recursive)
! growth -= info->size;
/* COMDAT functions are very often not shared across multiple units since they
come from various template instantiations. Take this into account. */
else if (DECL_COMDAT (node->decl) && !self_recursive
&& cgraph_can_remove_if_no_direct_calls_p (node))
! growth -= (info->size
* (100 - PARAM_VALUE (PARAM_COMDAT_SHARING_PROBABILITY)) + 50) / 100;
! info->estimated_growth = growth;
return growth;
}
*************** inline_read_summary (void)
*** 561,579 ****
struct cgraph_node *node;
struct inline_summary *info;
lto_cgraph_encoder_t encoder;
index = lto_input_uleb128 (ib);
encoder = file_data->cgraph_node_encoder;
node = lto_cgraph_encoder_deref (encoder, index);
info = inline_summary (node);
! node->global.estimated_stack_size
= info->estimated_self_stack_size = lto_input_uleb128 (ib);
! node->global.time = info->self_time = lto_input_uleb128 (ib);
info->time_inlining_benefit = lto_input_uleb128 (ib);
! node->global.size = info->self_size = lto_input_uleb128 (ib);
info->size_inlining_benefit = lto_input_uleb128 (ib);
! node->global.estimated_growth = INT_MIN;
}
lto_destroy_simple_input_block (file_data,
--- 603,627 ----
struct cgraph_node *node;
struct inline_summary *info;
lto_cgraph_encoder_t encoder;
+ struct bitpack_d bp;
index = lto_input_uleb128 (ib);
encoder = file_data->cgraph_node_encoder;
node = lto_cgraph_encoder_deref (encoder, index);
info = inline_summary (node);
! info->estimated_stack_size
= info->estimated_self_stack_size = lto_input_uleb128 (ib);
! info->time = info->self_time = lto_input_uleb128 (ib);
info->time_inlining_benefit = lto_input_uleb128 (ib);
! info->size = info->self_size = lto_input_uleb128 (ib);
info->size_inlining_benefit = lto_input_uleb128 (ib);
! info->estimated_growth = INT_MIN;
!
! bp = lto_input_bitpack (ib);
! info->inlinable = bp_unpack_value (&bp, 1);
! info->versionable = bp_unpack_value (&bp, 1);
! info->disregard_inline_limits = bp_unpack_value (&bp, 1);
}
lto_destroy_simple_input_block (file_data,
*************** inline_write_summary (cgraph_node_set se
*** 623,628 ****
--- 671,678 ----
if (node->analyzed)
{
struct inline_summary *info = inline_summary (node);
+ struct bitpack_d bp;
+
lto_output_uleb128_stream (ob->main_stream,
lto_cgraph_encoder_encode (encoder, node));
lto_output_sleb128_stream (ob->main_stream,
*************** inline_write_summary (cgraph_node_set se
*** 635,640 ****
--- 685,695 ----
info->self_time);
lto_output_sleb128_stream (ob->main_stream,
info->time_inlining_benefit);
+ bp = bitpack_create (ob->main_stream);
+ bp_pack_value (&bp, info->inlinable, 1);
+ bp_pack_value (&bp, info->versionable, 1);
+ bp_pack_value (&bp, info->disregard_inline_limits, 1);
+ lto_output_bitpack (&bp);
}
}
lto_destroy_simple_output_block (ob);
Index: tree-sra.c
===================================================================
*** tree-sra.c (revision 172431)
--- tree-sra.c (working copy)
*************** along with GCC; see the file COPYING3.
*** 91,96 ****
--- 91,97 ----
#include "dbgcnt.h"
#include "tree-inline.h"
#include "gimple-pretty-print.h"
+ #include "ipa-inline.h"
/* Enumeration of all aggregate reductions we can do. */
enum sra_mode { SRA_MODE_EARLY_IPA, /* early call regularization */
*************** ipa_sra_preliminary_function_checks (str
*** 4469,4475 ****
}
if ((DECL_COMDAT (node->decl) || DECL_EXTERNAL (node->decl))
! && node->global.size >= MAX_INLINE_INSNS_AUTO)
{
if (dump_file)
fprintf (dump_file, "Function too big to be made truly local.\n");
--- 4470,4476 ----
}
if ((DECL_COMDAT (node->decl) || DECL_EXTERNAL (node->decl))
! && inline_summary(node)->size >= MAX_INLINE_INSNS_AUTO)
{
if (dump_file)
fprintf (dump_file, "Function too big to be made truly local.\n");
Index: lto/lto.c
===================================================================
*** lto/lto.c (revision 172431)
--- lto/lto.c (working copy)
*************** lto_balanced_map (void)
*** 1030,1036 ****
if (partition_cgraph_node_p (node))
{
order[n_nodes++] = node;
! total_size += node->global.size;
}
}
free (postorder);
--- 1030,1036 ----
if (partition_cgraph_node_p (node))
{
order[n_nodes++] = node;
! total_size += inline_summary (node)->size;
}
}
free (postorder);
*************** lto_balanced_map (void)
*** 1049,1055 ****
{
if (!order[i]->aux)
add_cgraph_node_to_partition (partition, order[i]);
! total_size -= order[i]->global.size;
/* Once we added a new node to the partition, we also want to add
all referenced variables unless they was already added into some
--- 1049,1055 ----
{
if (!order[i]->aux)
add_cgraph_node_to_partition (partition, order[i]);
! total_size -= inline_summary (order[i])->size;
/* Once we added a new node to the partition, we also want to add
all referenced variables unless they was already added into some
Index: Makefile.in
===================================================================
*** Makefile.in (revision 172431)
--- Makefile.in (working copy)
*************** cgraph.o : cgraph.c $(CONFIG_H) $(SYSTEM
*** 2983,2989 ****
langhooks.h toplev.h $(DIAGNOSTIC_CORE_H) $(FLAGS_H) $(GGC_H) $(TARGET_H) $(CGRAPH_H) \
gt-cgraph.h output.h intl.h $(BASIC_BLOCK_H) debug.h $(HASHTAB_H) \
$(TREE_INLINE_H) $(TREE_DUMP_H) $(TREE_FLOW_H) cif-code.def \
! value-prof.h $(EXCEPT_H) $(IPA_UTILS_H) $(DIAGNOSTIC_CORE_H)
cgraphunit.o : cgraphunit.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) \
$(TREE_H) langhooks.h $(TREE_INLINE_H) toplev.h $(DIAGNOSTIC_CORE_H) $(FLAGS_H) $(GGC_H) \
$(TARGET_H) $(CGRAPH_H) intl.h pointer-set.h $(FUNCTION_H) $(GIMPLE_H) \
--- 2983,2990 ----
langhooks.h toplev.h $(DIAGNOSTIC_CORE_H) $(FLAGS_H) $(GGC_H) $(TARGET_H) $(CGRAPH_H) \
gt-cgraph.h output.h intl.h $(BASIC_BLOCK_H) debug.h $(HASHTAB_H) \
$(TREE_INLINE_H) $(TREE_DUMP_H) $(TREE_FLOW_H) cif-code.def \
! value-prof.h $(EXCEPT_H) $(IPA_UTILS_H) $(DIAGNOSTIC_CORE_H) \
! ipa-inline.h
cgraphunit.o : cgraphunit.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) \
$(TREE_H) langhooks.h $(TREE_INLINE_H) toplev.h $(DIAGNOSTIC_CORE_H) $(FLAGS_H) $(GGC_H) \
$(TARGET_H) $(CGRAPH_H) intl.h pointer-set.h $(FUNCTION_H) $(GIMPLE_H) \
*************** cgraphunit.o : cgraphunit.c $(CONFIG_H)
*** 2993,2999 ****
tree-pretty-print.h gimple-pretty-print.h
cgraphbuild.o : cgraphbuild.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) \
$(TREE_H) langhooks.h $(CGRAPH_H) intl.h pointer-set.h $(GIMPLE_H) \
! $(TREE_FLOW_H) $(TREE_PASS_H) $(IPA_UTILS_H) $(EXCEPT_H)
varpool.o : varpool.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) \
$(TREE_H) $(CGRAPH_H) langhooks.h $(DIAGNOSTIC_CORE_H) $(HASHTAB_H) \
$(GGC_H) $(TIMEVAR_H) debug.h $(TARGET_H) output.h $(GIMPLE_H) \
--- 2994,3001 ----
tree-pretty-print.h gimple-pretty-print.h
cgraphbuild.o : cgraphbuild.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) \
$(TREE_H) langhooks.h $(CGRAPH_H) intl.h pointer-set.h $(GIMPLE_H) \
! $(TREE_FLOW_H) $(TREE_PASS_H) $(IPA_UTILS_H) $(EXCEPT_H) \
! ipa-inline.h
varpool.o : varpool.c $(CONFIG_H) $(SYSTEM_H) coretypes.h $(TM_H) \
$(TREE_H) $(CGRAPH_H) langhooks.h $(DIAGNOSTIC_CORE_H) $(HASHTAB_H) \
$(GGC_H) $(TIMEVAR_H) debug.h $(TARGET_H) output.h $(GIMPLE_H) \