This is the mail archive of the
gcc-patches@gcc.gnu.org
mailing list for the GCC project.
[WPA PATCH 4/5] IPA-CP with function specialization
- From: Martin Jambor <mjambor at suse dot cz>
- To: GCC Patches <gcc-patches at gcc dot gnu dot org>
- Cc: Jan Hubicka <hubicka at ucw dot cz>
- Date: Wed, 05 Aug 2009 21:09:48 +0200
- Subject: [WPA PATCH 4/5] IPA-CP with function specialization
- References: <20090805190944.214311870@virgil.suse.cz>
This patch implements the function specialization for ipa-cp based on
benefit information gathered by the previous one. It stores small
structures called ipcp_opportunity into the fib heap and process them
one by one until the growth limit is reached. Each node has at most
one opportunity in the heap at any given time. The structure
identifies the associated node, index of the parameter that we
propagate a constant into, the constant and growth that this cloning
would result to.
The cost calculation certainly needs more tweaking and the overall
do/don't clone is too permissive. When identifying cloning
opportunities, we look for full-fledged constant propagation from all
known callers of !needed functions. If there are none, we examine
parameters one by one and look at all constants we can get from
different callers and pick the best option according to values
returned by function calculate_cost_for_cst_arg. At the moment, the
function drastically prefers cloning opportunities which we know to
lead to eradication of some basic blocks.
Seizing a cloning opportunity and carrying it is performed in function
seize_cloning_opportunity which is long but not really complex. It
identifies the edges to be redirected, then identifies all constants
that come along these edges, creates a virtual clone, identifies new
opportunities of all functions for which it is necessary.
Martin
2009-08-05 Martin Jambor <mjambor@suse.cz>
* ipa-prop.h (ipa_param_descriptor): New flag used_for_cloning.
(ipa_param_descriptor): New field ipa_cp_ignore, removed field
called_with_var_arguments, fields reordered.
(ipa_is_param_used_for_cloning): New function.
(ipa_mark_param_used_for_cloning): New function.
(ipa_set_called_with_variable_arg): Removed.
(ipa_set_ipa_cp_ignore): New function.
(ipa_is_called_with_var_arguments): Removed.
(ipa_is_ipa_cp_ignore): New function.
* ipa-cp.c: Deleted a lot of trailing blanks, added some missing
newlines. Include alloc-pool.h.
(struct ipcp_opportunity): New type.
(opportunity_pool): New variable.
(ipcp_init_cloned_node): Removed.
(ipcp_analyze_node): Move to the bottom of the file, check
ipcp_cloneable_function_p and call ipa_analyze_params_uses.
(ipcp_lat_is_const): Simplified.
(ipcp_lat_is_insertable): Removed.
(ipcp_lats_are_equal): Updated comments, removed a superfluous
comparison.
(ipcp_lattice_from_jfunc): Set lat->constant to error_mark_node for
BOTTOM type lattices.
(ipcp_versionable_function_p): Renamed to ipcp_cloneable_function_p.
Check body overwritability, dump reasons for refusal.
(ipcp_cloning_candidate_p): Check for ipa_cp_ignore flag instead for
checks for overwritability and versionability. Produce dumps only at
detailed dump level.
(ipcp_get_node_initial_lat_type): New function.
(ipcp_initialize_node_lattices): Get initial lattice type from
ipcp_get_node_initial_lat_type.
(ipcp_compute_node_scale): Calculate incoming edge count sum only when
required.
(ipcp_change_tops_to_bottom): Produce dumps only at detailed dump
level.
(ipcp_propagate_stage): Check ipa_is_ipa_cp_ignore instead of
ipa_is_called_with_var_arguments.
(ipcp_node_modifiable_p): Removed.
(ipcp_need_redirect_p): Don't look at the node we cloned from, use
param_used_for_cloning flag.
(ipcp_update_callgraph): Simplify.
(ipcp_const_param_count): Removed.
(ipcp_insert_stage): Removed.
(ipcp_estimate_cloning_cost): Removed.
(insert_opportunity_to_heap): New function.
(is_string_const): New function.
(calculate_cost_for_cst_arg): New function.
(ipcp_parameter_irrelevant_for_cost_p): New function.
(reevaluate_node_lattices): New function.
(insert_const_prop_opportunity_to_heap): New function.
(insert_best_opportunity_to_heap): New function.
(edge_from_removed_bb_p): New function.
(seize_cloning_opportunity): New function.
(ipcp_decisions_stage): New function.
(ipcp_driver): Call ipcp_decisions_stage instead of pcp_insert_stage.
(ipcp_generate_summary): Calculate jump functions directly instead of
calling ipcp_init_stage.
(cgraph_gate_cp): Renamed to gate_ipa_cp.
* Makefile.in (ipa-cp.o): Add alloc-pool.h to dependencies.
* cgraph.c (cgraph_create_virtual_clone): Assert node->clone_of.
(verify_cgraph_node): Do not check that there is an edge for each call
statement.
(create_missing_cg_edge): New function.
(copy_bb): Moved creating missing edges to create_missing_cg_edge
which is called.
(expand_call_inline): Handle missing call graph edges gracefully.
Index: icln/gcc/ipa-cp.c
===================================================================
--- icln.orig/gcc/ipa-cp.c 2009-07-31 20:01:19.000000000 +0200
+++ icln/gcc/ipa-cp.c 2009-07-31 20:19:15.000000000 +0200
@@ -1,19 +1,19 @@
/* Interprocedural constant propagation
Copyright (C) 2005, 2006, 2007, 2008, 2009 Free Software Foundation, Inc.
Contributed by Razya Ladelsky <RAZYA@il.ibm.com>
-
+
This file is part of GCC.
-
+
GCC is free software; you can redistribute it and/or modify it under
the terms of the GNU General Public License as published by the Free
Software Foundation; either version 3, or (at your option) any later
version.
-
+
GCC is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License
for more details.
-
+
You should have received a copy of the GNU General Public License
along with GCC; see the file COPYING3. If not see
<http://www.gnu.org/licenses/>. */
@@ -27,7 +27,7 @@ along with GCC; see the file COPYING3.
{
printf ("value is %d",y);
}
-
+
int f (int x)
{
g (x);
@@ -43,33 +43,33 @@ along with GCC; see the file COPYING3.
f (3);
h (3);
}
-
-
+
+
The IPCP algorithm will find that g's formal argument y is always called
with the value 3.
The algorithm used is based on "Interprocedural Constant Propagation", by
Challahan David, Keith D Cooper, Ken Kennedy, Linda Torczon, Comp86, pg
152-161
-
+
The optimization is divided into three stages:
First stage - intraprocedural analysis
=======================================
This phase computes jump_function and modification flags.
-
+
A jump function for a callsite represents the values passed as an actual
arguments of a given callsite. There are three types of values:
Pass through - the caller's formal parameter is passed as an actual argument.
Constant - a constant is passed as an actual argument.
Unknown - neither of the above.
-
+
The jump function info, ipa_jump_func, is stored in ipa_edge_args
structure (defined in ipa_prop.h and pointed to by cgraph_node->aux)
modified_flags are defined in ipa_node_params structure
(defined in ipa_prop.h and pointed to by cgraph_edge->aux).
-
- -ipcp_init_stage() is the first stage driver.
+
+ -ipcp_generate_summary() is the first stage driver.
Second stage - interprocedural analysis
========================================
@@ -79,10 +79,10 @@ along with GCC; see the file COPYING3.
TOP - unknown.
BOTTOM - non constant.
CONSTANT - constant value.
-
+
Lattice describing a formal parameter p will have a constant value if all
callsites invoking this function have the same constant value passed to p.
-
+
The lattices are stored in ipcp_lattice which is itself in ipa_node_params
structure (defined in ipa_prop.h and pointed to by cgraph_edge->aux).
@@ -115,12 +115,13 @@ along with GCC; see the file COPYING3.
and many calls redirected back to fit the description above.
-ipcp_insert_stage() is the third phase driver.
-
+
*/
#include "config.h"
#include "system.h"
#include "coretypes.h"
+#include "alloc-pool.h"
#include "tree.h"
#include "target.h"
#include "cgraph.h"
@@ -135,22 +136,42 @@ along with GCC; see the file COPYING3.
#include "fibheap.h"
#include "params.h"
+/* Structure representing the best cloning opportunity of a cgraph node. */
+struct ipcp_opportunity
+{
+ /* Callers of what node should be considered. */
+ struct cgraph_node *node;
+ /* What is the constant value of the determining argument. */
+ tree cst;
+ /* Index of the determining argument. */
+ int parm_index;
+ /* Size increase after cloning. */
+ int growth;
+};
+
+/* allocation pool for opportunity. */
+
+static alloc_pool opportunity_pool;
+
/* Number of functions identified as candidates for cloning. When not cloning
we can simplify iterate stage not forcing it to go through the decision
on what is profitable and what not. */
+
static int n_cloning_candidates;
/* Maximal count found in program. */
+
static gcov_type max_count;
/* Cgraph nodes that has been completely replaced by cloning during iterate
* stage and will be removed after ipcp is finished. */
+
static bitmap dead_nodes;
static void ipcp_print_profile_data (FILE *);
-static void ipcp_function_scale_print (FILE *);
/* Get the original node field of ipa_node_params associated with node NODE. */
+
static inline struct cgraph_node *
ipcp_get_orig_node (struct cgraph_node *node)
{
@@ -158,35 +179,15 @@ ipcp_get_orig_node (struct cgraph_node *
}
/* Return true if NODE describes a cloned/versioned function. */
+
static inline bool
ipcp_node_is_clone (struct cgraph_node *node)
{
return (ipcp_get_orig_node (node) != NULL);
}
-/* Create ipa_node_params and its data structures for NEW_NODE. Set ORIG_NODE
- as the ipcp_orig_node field in ipa_node_params. */
-static void
-ipcp_init_cloned_node (struct cgraph_node *orig_node,
- struct cgraph_node *new_node)
-{
- ipa_check_create_node_params ();
- ipa_initialize_node_params (new_node);
- IPA_NODE_REF (new_node)->ipcp_orig_node = orig_node;
-}
-
-/* Perform intraprocedrual analysis needed for ipcp. */
-static void
-ipcp_analyze_node (struct cgraph_node *node)
-{
- /* Unreachable nodes should have been eliminated before ipcp. */
- gcc_assert (node->needed || node->reachable);
-
- ipa_initialize_node_params (node);
- ipa_detect_param_modifications (node);
-}
-
/* Return scale for NODE. */
+
static inline gcov_type
ipcp_get_node_scale (struct cgraph_node *node)
{
@@ -194,6 +195,7 @@ ipcp_get_node_scale (struct cgraph_node
}
/* Set COUNT as scale for NODE. */
+
static inline void
ipcp_set_node_scale (struct cgraph_node *node, gcov_type count)
{
@@ -201,30 +203,20 @@ ipcp_set_node_scale (struct cgraph_node
}
/* Return whether LAT is a constant lattice. */
+
static inline bool
ipcp_lat_is_const (struct ipcp_lattice *lat)
{
- if (lat->type == IPA_CONST_VALUE)
- return true;
- else
- return false;
+ return (lat->type == IPA_CONST_VALUE);
}
-/* Return whether LAT is a constant lattice that ipa-cp can actually insert
- into the code (i.e. constants excluding member pointers and pointers). */
-static inline bool
-ipcp_lat_is_insertable (struct ipcp_lattice *lat)
-{
- return lat->type == IPA_CONST_VALUE;
-}
+/* Return true if LAT1 and LAT2 are equal. Both LAT1 and LAT2 need to be
+ constant lattices. */
-/* Return true if LAT1 and LAT2 are equal. */
static inline bool
ipcp_lats_are_equal (struct ipcp_lattice *lat1, struct ipcp_lattice *lat2)
{
gcc_assert (ipcp_lat_is_const (lat1) && ipcp_lat_is_const (lat2));
- if (lat1->type != lat2->type)
- return false;
if (operand_equal_p (lat1->constant, lat2->constant, 0))
return true;
@@ -237,6 +229,7 @@ ipcp_lats_are_equal (struct ipcp_lattice
Meet (IPA_TOP,x) = x
Meet (const_a,const_b) = IPA_BOTTOM, if const_a != const_b.
MEET (const_a,const_b) = const_a, if const_a == const_b.*/
+
static void
ipa_lattice_meet (struct ipcp_lattice *res, struct ipcp_lattice *lat1,
struct ipcp_lattice *lat2)
@@ -269,6 +262,7 @@ ipa_lattice_meet (struct ipcp_lattice *r
/* Return the lattice corresponding to the Ith formal parameter of the function
described by INFO. */
+
static inline struct ipcp_lattice *
ipcp_get_lattice (struct ipa_node_params *info, int i)
{
@@ -278,6 +272,7 @@ ipcp_get_lattice (struct ipa_node_params
/* Given the jump function JFUNC, compute the lattice LAT that describes the
value coming down the callsite. INFO describes the caller node so that
pass-through jump functions can be evaluated. */
+
static void
ipcp_lattice_from_jfunc (struct ipa_node_params *info, struct ipcp_lattice *lat,
struct ipa_jump_func *jfunc)
@@ -331,7 +326,10 @@ ipcp_lattice_from_jfunc (struct ipa_node
lat->constant = build_fold_addr_expr (t);
}
else
- lat->type = IPA_BOTTOM;
+ {
+ lat->type = IPA_BOTTOM;
+ lat->constant = error_mark_node;
+ }
}
/* True when OLD_LAT and NEW_LAT values are not the same. */
@@ -351,6 +349,7 @@ ipcp_lattice_changed (struct ipcp_lattic
}
/* Print all ipcp_lattices of all functions to F. */
+
static void
ipcp_print_all_lattices (FILE * f)
{
@@ -386,21 +385,48 @@ ipcp_print_all_lattices (FILE * f)
}
}
-/* Return true if ipcp algorithms would allow cloning NODE. */
+/* Return true if it is technically possible for ipa-cp to clone NODE, which
+ must never be a clone. */
static bool
-ipcp_versionable_function_p (struct cgraph_node *node)
+ipcp_cloneable_function_p (struct cgraph_node *node)
{
tree decl = node->decl;
basic_block bb;
+ gcc_assert (!ipcp_node_is_clone (node));
+
+ if (!node->analyzed)
+ return false;
+
+ if (cgraph_function_body_availability (node) <= AVAIL_OVERWRITABLE)
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file,
+ "Not considering %s for cloning; body is overwrittable.\n",
+ cgraph_node_name (node));
+ return false;
+ }
+
/* There are a number of generic reasons functions cannot be versioned. */
if (!tree_versionable_function_p (decl))
- return false;
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file,
+ "Not considering %s for cloning; body is not versionable.\n",
+ cgraph_node_name (node));
+ return false;
+ }
/* Removing arguments doesn't work if the function takes varargs. */
if (DECL_STRUCT_FUNCTION (decl)->stdarg)
- return false;
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file,
+ "Not considering %s for cloning; function uses stdargs.\n",
+ cgraph_node_name (node));
+ return false;
+ }
/* Removing arguments doesn't work if we use __builtin_apply_args. */
FOR_EACH_BB_FN (bb, DECL_STRUCT_FUNCTION (decl))
@@ -418,43 +444,36 @@ ipcp_versionable_function_p (struct cgra
continue;
if (DECL_BUILT_IN_CLASS (t) == BUILT_IN_NORMAL
&& DECL_FUNCTION_CODE (t) == BUILT_IN_APPLY_ARGS)
- return false;
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file,
+ "Not considering %s for cloning; "
+ "function uses __builtin_apply_args.\n",
+ cgraph_node_name (node));
+ return false;
+ }
}
}
return true;
}
-/* Return true if this NODE is viable candidate for cloning. */
+
+/* Return true if this NODE (described by INFO) is a potential candidate for
+ cloning. */
+
static bool
-ipcp_cloning_candidate_p (struct cgraph_node *node)
+ipcp_cloning_candidate_p (struct cgraph_node *node,
+ struct ipa_node_params *info)
{
int n_calls = 0;
int n_hot_calls = 0;
gcov_type direct_call_sum = 0;
struct cgraph_edge *e;
- /* We never clone functions that are not visible from outside.
- FIXME: in future we should clone such functions when they are called with
- different constants, but current ipcp implementation is not good on this.
- */
- if (!node->needed || !node->analyzed)
+ if (ipa_is_ipa_cp_ignore (info))
return false;
- if (cgraph_function_body_availability (node) <= AVAIL_OVERWRITABLE)
- {
- if (dump_file)
- fprintf (dump_file, "Not considering %s for cloning; body is overwrittable.\n",
- cgraph_node_name (node));
- return false;
- }
- if (!ipcp_versionable_function_p (node))
- {
- if (dump_file)
- fprintf (dump_file, "Not considering %s for cloning; body is not versionable.\n",
- cgraph_node_name (node));
- return false;
- }
for (e = node->callers; e; e = e->next_caller)
{
direct_call_sum += e->count;
@@ -462,34 +481,30 @@ ipcp_cloning_candidate_p (struct cgraph_
if (cgraph_maybe_hot_edge_p (e))
n_hot_calls ++;
}
-
+
if (!n_calls)
{
- if (dump_file)
- fprintf (dump_file, "Not considering %s for cloning; no direct calls.\n",
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file,
+ "Not considering %s for cloning; no direct calls.\n",
cgraph_node_name (node));
return false;
}
+
if (node->local.inline_summary.self_size < n_calls)
{
- if (dump_file)
- fprintf (dump_file, "Considering %s for cloning; code would shrink.\n",
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file,
+ "Considering %s for cloning; code might shrink.\n",
cgraph_node_name (node));
return true;
- }
-
- if (!flag_ipa_cp_clone)
- {
- if (dump_file)
- fprintf (dump_file, "Not considering %s for cloning; -fipa-cp-clone disabled.\n",
- cgraph_node_name (node));
- return false;
}
- if (!optimize_function_for_speed_p (DECL_STRUCT_FUNCTION (node->decl)))
+ if (!flag_ipa_cp_clone)
{
- if (dump_file)
- fprintf (dump_file, "Not considering %s for cloning; optimizing it for size.\n",
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file,
+ "Not considering %s for cloning; -fipa-cp-clone disabled.\n",
cgraph_node_name (node));
return false;
}
@@ -501,52 +516,72 @@ ipcp_cloning_candidate_p (struct cgraph_
{
if (direct_call_sum > node->count * 90 / 100)
{
- if (dump_file)
- fprintf (dump_file, "Considering %s for cloning; usually called directly.\n",
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file,
+ "Considering %s for cloning; usually called directly.\n",
cgraph_node_name (node));
return true;
}
}
+
if (!n_hot_calls)
{
- if (dump_file)
+ if (dump_file && (dump_flags & TDF_DETAILS))
fprintf (dump_file, "Not considering %s for cloning; no hot calls.\n",
cgraph_node_name (node));
return false;
}
- if (dump_file)
+
+ if (dump_file && (dump_flags & TDF_DETAILS))
fprintf (dump_file, "Considering %s for cloning.\n",
cgraph_node_name (node));
return true;
}
-/* Initialize ipcp_lattices array. The lattices corresponding to supported
- types (integers, real types and Fortran constants defined as const_decls)
- are initialized to IPA_TOP, the rest of them to IPA_BOTTOM. */
-static void
-ipcp_initialize_node_lattices (struct cgraph_node *node)
+/* Return the lattice type to start with for parameters of function
+ corresponding to cgraph node NODE according to its properties. */
+
+static enum ipa_lattice_type
+ipcp_get_node_initial_lat_type (struct cgraph_node *node,
+ struct ipa_node_params *info)
{
- int i;
- struct ipa_node_params *info = IPA_NODE_REF (node);
enum ipa_lattice_type type;
- if (ipa_is_called_with_var_arguments (info))
+ if (ipa_is_ipa_cp_ignore (info))
type = IPA_BOTTOM;
else if (!node->needed)
type = IPA_TOP;
/* When cloning is allowed, we can assume that externally visible functions
are not called. We will compensate this by cloning later. */
- else if (ipcp_cloning_candidate_p (node))
+ else if (ipcp_cloning_candidate_p (node, info))
type = IPA_TOP, n_cloning_candidates ++;
else
type = IPA_BOTTOM;
+ return type;
+}
+
+/* Initialize ipcp_lattices array in such a way that the lattices of functions
+ that ipa-cp cannot operate on or those that would have to be cloned and
+ should not be are set to IPA_BOTTOM, the rest to IPA_TOP. Return true in
+ the latter case and false in the formar. */
+
+static bool
+ipcp_initialize_node_lattices (struct cgraph_node *node)
+{
+ struct ipa_node_params *info = IPA_NODE_REF (node);
+ enum ipa_lattice_type type = ipcp_get_node_initial_lat_type (node, info);
+ int i;
+
for (i = 0; i < ipa_get_param_count (info) ; i++)
ipcp_get_lattice (info, i)->type = type;
+
+ return type == IPA_TOP;
}
-/* build INTEGER_CST tree with type TREE_TYPE and value according to LAT.
+/* Build a constant tree with type TREE_TYPE and value according to LAT.
Return the tree. */
+
static tree
build_const_val (struct ipcp_lattice *lat, tree tree_type)
{
@@ -568,62 +603,27 @@ build_const_val (struct ipcp_lattice *la
/* Compute the proper scale for NODE. It is the ratio between the number of
direct calls (represented on the incoming cgraph_edges) and sum of all
invocations of NODE (represented as count in cgraph_node). */
+
static void
ipcp_compute_node_scale (struct cgraph_node *node)
{
- gcov_type sum;
- struct cgraph_edge *cs;
-
- sum = 0;
- /* Compute sum of all counts of callers. */
- for (cs = node->callers; cs != NULL; cs = cs->next_caller)
- sum += cs->count;
if (node->count == 0)
ipcp_set_node_scale (node, 0);
else
- ipcp_set_node_scale (node, sum * REG_BR_PROB_BASE / node->count);
-}
-
-/* Initialization and computation of IPCP data structures. This is the initial
- intraprocedural analysis of functions, which gathers information to be
- propagated later on. */
-static void
-ipcp_init_stage (void)
-{
- struct cgraph_node *node;
- struct cgraph_edge *cs;
-
- for (node = cgraph_nodes; node; node = node->next)
- if (node->analyzed)
- ipcp_analyze_node (node);
- for (node = cgraph_nodes; node; node = node->next)
{
- if (!node->analyzed)
- continue;
- /* building jump functions */
- for (cs = node->callees; cs; cs = cs->next_callee)
- {
- if (!cs->callee->analyzed)
- continue;
- ipa_count_arguments (cs);
- if (ipa_get_cs_argument_count (IPA_EDGE_REF (cs))
- != ipa_get_param_count (IPA_NODE_REF (cs->callee)))
- {
- /* Handle cases of functions with
- a variable number of parameters. */
- ipa_set_called_with_variable_arg (IPA_NODE_REF (cs->callee));
- if (flag_indirect_inlining)
- ipa_compute_jump_functions (cs);
- }
- else
- ipa_compute_jump_functions (cs);
- }
+ struct cgraph_edge *cs;
+ gcov_type sum = 0;
+ /* Compute sum of all counts of callers. */
+ for (cs = node->callers; cs != NULL; cs = cs->next_caller)
+ sum += cs->count;
+ ipcp_set_node_scale (node, sum * REG_BR_PROB_BASE / node->count);
}
}
/* Return true if there are some formal parameters whose value is IPA_TOP (in
the whole compilation unit). Change their values to IPA_BOTTOM, since they
most probably get their values from outside of this compilation unit. */
+
static bool
ipcp_change_tops_to_bottom (void)
{
@@ -642,7 +642,7 @@ ipcp_change_tops_to_bottom (void)
if (lat->type == IPA_TOP)
{
prop_again = true;
- if (dump_file)
+ if (dump_file && (dump_flags & TDF_DETAILS))
{
fprintf (dump_file, "Forcing param ");
print_generic_expr (dump_file, ipa_get_param (info, i), 0);
@@ -658,6 +658,7 @@ ipcp_change_tops_to_bottom (void)
/* Interprocedural analysis. The algorithm propagates constants from the
caller's parameters to the callee's arguments. */
+
static void
ipcp_propagate_stage (void)
{
@@ -685,7 +686,7 @@ ipcp_propagate_stage (void)
struct ipa_node_params *callee_info = IPA_NODE_REF (cs->callee);
struct ipa_edge_args *args = IPA_EDGE_REF (cs);
- if (ipa_is_called_with_var_arguments (callee_info))
+ if (ipa_is_ipa_cp_ignore (callee_info))
continue;
count = ipa_get_cs_argument_count (args);
@@ -706,8 +707,26 @@ ipcp_propagate_stage (void)
}
}
+/* Print count scale data structures. */
+
+static void
+ipcp_function_scale_print (FILE * f)
+{
+ struct cgraph_node *node;
+
+ for (node = cgraph_nodes; node; node = node->next)
+ {
+ if (!node->analyzed)
+ continue;
+ fprintf (f, "printing scale for %s: ", cgraph_node_name (node));
+ fprintf (f, "value is " HOST_WIDE_INT_PRINT_DEC
+ " \n", (HOST_WIDE_INT) ipcp_get_node_scale (node));
+ }
+}
+
/* Call the constant propagation algorithm and re-call it if necessary
(if there are undetermined values left). */
+
static void
ipcp_iterate_stage (void)
{
@@ -744,33 +763,8 @@ ipcp_iterate_stage (void)
}
}
-/* Check conditions to forbid constant insertion to function described by
- NODE. */
-static inline bool
-ipcp_node_modifiable_p (struct cgraph_node *node)
-{
- /* Once we will be able to do in-place replacement, we can be more
- lax here. */
- return ipcp_versionable_function_p (node);
-}
-
-/* Print count scale data structures. */
-static void
-ipcp_function_scale_print (FILE * f)
-{
- struct cgraph_node *node;
-
- for (node = cgraph_nodes; node; node = node->next)
- {
- if (!node->analyzed)
- continue;
- fprintf (f, "printing scale for %s: ", cgraph_node_name (node));
- fprintf (f, "value is " HOST_WIDE_INT_PRINT_DEC
- " \n", (HOST_WIDE_INT) ipcp_get_node_scale (node));
- }
-}
-
/* Print counts of all cgraph nodes. */
+
static void
ipcp_print_func_profile_counts (FILE * f)
{
@@ -785,6 +779,7 @@ ipcp_print_func_profile_counts (FILE * f
}
/* Print counts of all cgraph edges. */
+
static void
ipcp_print_call_profile_counts (FILE * f)
{
@@ -804,6 +799,7 @@ ipcp_print_call_profile_counts (FILE * f
}
/* Print profile info for all functions. */
+
static void
ipcp_print_profile_data (FILE * f)
{
@@ -817,6 +813,7 @@ ipcp_print_profile_data (FILE * f)
processed by versioning, which operates according to the flags set.
PARM_TREE is the formal parameter found to be constant. LAT represents the
constant. */
+
static struct ipa_replace_map *
ipcp_create_replace_map (tree parm_tree, struct ipcp_lattice *lat)
{
@@ -827,7 +824,7 @@ ipcp_create_replace_map (tree parm_tree,
const_val = build_const_val (lat, TREE_TYPE (parm_tree));
if (dump_file)
{
- fprintf (dump_file, " replacing param ");
+ fprintf (dump_file, " replacing param ");
print_generic_expr (dump_file, parm_tree, 0);
fprintf (dump_file, " with const ");
print_generic_expr (dump_file, const_val, 0);
@@ -843,39 +840,33 @@ ipcp_create_replace_map (tree parm_tree,
/* Return true if this callsite should be redirected to the original callee
(instead of the cloned one). */
+
static bool
ipcp_need_redirect_p (struct cgraph_edge *cs)
{
- struct ipa_node_params *orig_callee_info;
+ struct ipa_node_params *callee_info;
int i, count;
- struct ipa_jump_func *jump_func;
- struct cgraph_node *node = cs->callee, *orig;
- if (!n_cloning_candidates)
+ if (!n_cloning_candidates || cs->caller == cs->callee)
return false;
-
- if ((orig = ipcp_get_orig_node (node)) != NULL)
- node = orig;
if (ipcp_get_orig_node (cs->caller))
return false;
- orig_callee_info = IPA_NODE_REF (node);
- count = ipa_get_param_count (orig_callee_info);
+ callee_info = IPA_NODE_REF (cs->callee);
+ count = ipa_get_param_count (callee_info);
for (i = 0; i < count; i++)
- {
- struct ipcp_lattice *lat = ipcp_get_lattice (orig_callee_info, i);
- if (ipcp_lat_is_const (lat))
- {
- jump_func = ipa_get_ith_jump_func (IPA_EDGE_REF (cs), i);
- if (jump_func->type != IPA_JF_CONST)
- return true;
- }
- }
-
+ if (ipa_is_param_used_for_cloning (callee_info, i))
+ {
+ struct ipa_jump_func *jump_func;
+ jump_func = ipa_get_ith_jump_func (IPA_EDGE_REF (cs), i);
+ if (jump_func->type != IPA_JF_CONST)
+ return true;
+ }
return false;
}
/* Fix the callsites and the call graph after function cloning was done. */
+
static void
ipcp_update_callgraph (void)
{
@@ -884,33 +875,13 @@ ipcp_update_callgraph (void)
for (node = cgraph_nodes; node; node = node->next)
if (node->analyzed && ipcp_node_is_clone (node))
{
- bitmap args_to_skip = BITMAP_ALLOC (NULL);
struct cgraph_node *orig_node = ipcp_get_orig_node (node);
- struct ipa_node_params *info = IPA_NODE_REF (orig_node);
- int i, count = ipa_get_param_count (info);
struct cgraph_edge *cs, *next;
- for (i = 0; i < count; i++)
- {
- struct ipcp_lattice *lat = ipcp_get_lattice (info, i);
- tree parm_tree = ipa_get_param (info, i);
-
- /* We can proactively remove obviously unused arguments. */
- if (is_gimple_reg (parm_tree)
- && !gimple_default_def (DECL_STRUCT_FUNCTION (orig_node->decl),
- parm_tree))
- {
- bitmap_set_bit (args_to_skip, i);
- continue;
- }
-
- if (lat->type == IPA_CONST_VALUE)
- bitmap_set_bit (args_to_skip, i);
- }
for (cs = node->callers; cs; cs = next)
{
next = cs->next_caller;
- if (!ipcp_node_is_clone (cs->caller) && ipcp_need_redirect_p (cs))
+ if (ipcp_need_redirect_p (cs))
cgraph_redirect_edge_callee (cs, orig_node);
}
}
@@ -918,6 +889,7 @@ ipcp_update_callgraph (void)
/* Update profiling info for versioned functions and the functions they were
versioned from. */
+
static void
ipcp_update_profiling (void)
{
@@ -943,140 +915,583 @@ ipcp_update_profiling (void)
}
}
-/* If NODE was cloned, how much would program grow? */
-static long
-ipcp_estimate_growth (struct cgraph_node *node)
-{
- struct cgraph_edge *cs;
- int redirectable_node_callers = 0;
- int removable_args = 0;
- bool need_original = node->needed;
- struct ipa_node_params *info;
- int i, count;
- int growth;
+/* Create an opportunity structure asdescribed by the parameters and insert it
+ into the HEAP. */
- for (cs = node->callers; cs != NULL; cs = cs->next_caller)
- if (cs->caller == node || !ipcp_need_redirect_p (cs))
- redirectable_node_callers++;
- else
- need_original = true;
-
- /* If we will be able to fully replace orignal node, we never increase
- program size. */
- if (!need_original)
- return 0;
+static void
+insert_opportunity_to_heap (fibheap_t heap, int cost, struct cgraph_node *node,
+ int parm_index, tree cst, int growth)
+{
+ struct ipcp_opportunity *opp;
- info = IPA_NODE_REF (node);
- count = ipa_get_param_count (info);
- for (i = 0; i < count; i++)
+ if (dump_file)
{
- struct ipcp_lattice *lat = ipcp_get_lattice (info, i);
- tree parm_tree = ipa_get_param (info, i);
-
- /* We can proactively remove obviously unused arguments. */
- if (is_gimple_reg (parm_tree)
- && !gimple_default_def (DECL_STRUCT_FUNCTION (node->decl),
- parm_tree))
- removable_args++;
-
- if (lat->type == IPA_CONST_VALUE)
- removable_args++;
+ fprintf (dump_file, "Inserting opportunity to clone %s:\n",
+ cgraph_node_name (node));
+ fprintf (dump_file, " cost: %i, index: %i, growth: %i, cst: ",
+ cost, parm_index, growth);
+ print_generic_expr (dump_file, cst, 0);
+ fprintf (dump_file, "\n");
}
- /* We make just very simple estimate of savings for removal of operand from
- call site. Precise cost is dificult to get, as our size metric counts
- constants and moves as free. Generally we are looking for cases that
- small function is called very many times. */
- growth = node->local.inline_summary.self_size
- - removable_args * redirectable_node_callers;
- if (growth < 0)
- return 0;
- return growth;
+ gcc_assert (cst);
+ gcc_assert (parm_index >= 0);
+ opp = (struct ipcp_opportunity *) pool_alloc (opportunity_pool);
+ opp->node = node;
+ opp->cst = cst;
+ opp->parm_index = parm_index;
+ opp->growth = growth;
+ node->aux = fibheap_insert (heap, cost, opp);
}
+/* Return true iff T is a string constant. */
-/* Estimate cost of cloning NODE. */
-static long
-ipcp_estimate_cloning_cost (struct cgraph_node *node)
+static bool
+is_string_const (tree t)
{
- int freq_sum = 1;
- gcov_type count_sum = 1;
- struct cgraph_edge *e;
- int cost;
-
- cost = ipcp_estimate_growth (node) * 1000;
- if (!cost)
+ if (TREE_CODE (t) == ADDR_EXPR)
{
- if (dump_file)
- fprintf (dump_file, "Versioning of %s will save code size\n",
- cgraph_node_name (node));
- return 0;
+ t = TREE_OPERAND (t, 0);
+ if (TREE_CODE (t) == ARRAY_REF)
+ t = TREE_OPERAND (t, 0);
}
- for (e = node->callers; e; e = e->next_caller)
- if (!bitmap_bit_p (dead_nodes, e->caller->uid)
- && !ipcp_need_redirect_p (e))
- {
- count_sum += e->count;
- freq_sum += e->frequency + 1;
- }
-
- if (max_count)
- cost /= count_sum * 1000 / max_count + 1;
- else
- cost /= freq_sum * 1000 / REG_BR_PROB_BASE + 1;
- if (dump_file)
- fprintf (dump_file, "Cost of versioning %s is %i, (size: %i, freq: %i)\n",
- cgraph_node_name (node), cost, node->local.inline_summary.self_size,
- freq_sum);
- return cost + 1;
+ return TREE_CODE (t) == STRING_CST;
}
-/* Return number of live constant parameters. */
+/* Calculate cost of cloning opportunity for NODE with parameter at PARM_INDEX
+ being constant CST. COUNT_SUM adn FREQ_SUM are sums of counts and
+ frequencies of incoming cgraph edges that would be redirected. Store how
+ much the program would grow into *GROWTH. */
+
static int
-ipcp_const_param_count (struct cgraph_node *node)
-{
- int const_param = 0;
- struct ipa_node_params *info = IPA_NODE_REF (node);
- int count = ipa_get_param_count (info);
- int i;
+calculate_cost_for_cst_arg (struct ipa_node_params *info,
+ struct cgraph_node *node,
+ int parm_index, tree cst, int caller_count,
+ int count_sum, int freq_sum, int *growth)
+{
+ unsigned i;
+ int j, b_count = ipa_get_number_of_cond_uses_benefits (info, parm_index);
+ bitmap removed;
+ bitmap_iterator bi;
+ int cost, saved_time, size = node->local.inline_summary.self_size;
- for (i = 0; i < count; i++)
+ if (b_count == 0)
{
- struct ipcp_lattice *lat = ipcp_get_lattice (info, i);
- tree parm_tree = ipa_get_param (info, i);
- if (ipcp_lat_is_insertable (lat)
- /* Do not count obviously unused arguments. */
- && (!is_gimple_reg (parm_tree)
- || gimple_default_def (DECL_STRUCT_FUNCTION (node->decl),
- parm_tree)))
- const_param++;
+ if (ipa_is_param_cond_used (info, parm_index))
+ cost = size / 2;
+ else
+ cost = size;
+ cost -= caller_count;
+ cost *= 1000;
+ if (max_count)
+ cost /= count_sum * 1000 / max_count + 1;
+ else
+ cost /= freq_sum * 1000 / REG_BR_PROB_BASE + 1;
+ *growth = size;
+ return cost + 1;
}
- return const_param;
-}
-/* Propagate the constant parameters found by ipcp_iterate_stage()
- to the function's code. */
-static void
-ipcp_insert_stage (void)
-{
- struct cgraph_node *node, *node1 = NULL;
- int i;
- VEC (cgraph_edge_p, heap) * redirect_callers;
- VEC (ipa_replace_map_p,gc)* replace_trees;
- int node_callers, count;
- tree parm_tree;
- struct ipa_replace_map *replace_param;
- fibheap_t heap;
+ removed = BITMAP_ALLOC (NULL);
+
+ for (j = 0; j< b_count; j++)
+ {
+ struct ipa_param_cond_benefit* ben;
+
+ ben = ipa_get_parm_cond_benefit (info, parm_index, j);
+ if (integer_nonzerop (fold_build2 (ben->oper, integer_type_node,
+ cst, ben->cst)))
+ bitmap_ior_into (removed, ben->true_deleted_bbs);
+ else
+ bitmap_ior_into (removed, ben->false_deleted_bbs);
+ }
+
+ saved_time = 0;
+ EXECUTE_IF_SET_IN_BITMAP (removed, 0, i, bi)
+ {
+ saved_time += VEC_index (int, info->bb_times, i);
+ size -= VEC_index (int, info->bb_sizes, i);
+ }
+ BITMAP_FREE (removed);
+ gcc_assert (size >= 0);
+
+ cost = - ((saved_time * 1000) / (size + 1));
+ if (max_count)
+ cost *= count_sum * 1000 / max_count + 1;
+ else
+ cost *= freq_sum * 1000 / REG_BR_PROB_BASE + 1;
+ *growth = size;
+ return cost - 1;
+}
+
+/* Return true iff Ith parameter of function described by NODE and INFO should
+ not be considered for the purposes of IPA-CP. This means that it is either
+ obviously unused or it has already been dealt with and removed (either
+ because it is unused or constant) in this particular clone. */
+
+static inline bool
+ipcp_parameter_irrelevant_for_cost_p (struct ipa_node_params *info, int i)
+{
+/* FIXME: Include also || DECL_PRESERVE_P (parm) once the attribute used for
+ params patch is in. */
+
+ return (ipa_is_param_used_for_cloning (info, i)
+ || ipa_is_param_unused (info, i));
+}
+
+/* Re-evaluate parameter lattices of NODE described by INFO. Return index of a
+ constant lattice if there is any or -1 otherwise. */
+
+static int
+reevaluate_node_lattices (struct cgraph_node *node,
+ struct ipa_node_params *info)
+{
+ int i, parm_count = ipa_get_param_count (info);
+ enum ipa_lattice_type init_type = ipcp_get_node_initial_lat_type (node, info);
+ int res = -1;
+
+ if (init_type == IPA_BOTTOM)
+ return -1;
+
+ for (i = 0; i < parm_count; i++)
+ {
+ struct ipcp_lattice *dest_lat;
+ struct cgraph_edge *cs;
+
+ if (ipcp_parameter_irrelevant_for_cost_p (info, i))
+ continue;
+
+ dest_lat = ipcp_get_lattice (info, i);
+ dest_lat->type = init_type;
+
+ for (cs = node->callers; cs; cs = cs->next_caller)
+ if (!bitmap_bit_p (dead_nodes, cs->caller->uid))
+ {
+ struct ipa_node_params *caller_info = IPA_NODE_REF (cs->caller);
+ struct ipa_edge_args *args = IPA_EDGE_REF (cs);
+ struct ipcp_lattice inc_lat = { IPA_BOTTOM, NULL };
+ struct ipa_jump_func *jump_func;
+
+ jump_func = ipa_get_ith_jump_func (args, i);
+ ipcp_lattice_from_jfunc (caller_info, &inc_lat, jump_func);
+ ipa_lattice_meet (dest_lat, dest_lat, &inc_lat);
+ }
+
+ if (ipcp_lat_is_const (dest_lat))
+ res = i;
+ }
+
+ for (i = 0; i < parm_count; i++)
+ {
+ struct ipcp_lattice *dest_lat = ipcp_get_lattice (info, i);
+
+ if (dest_lat->type == IPA_TOP)
+ dest_lat->type = IPA_BOTTOM;
+ }
+
+ return res;
+}
+
+/* Determine if we can do full iap constant propagation without creating an
+ extra clone and if so, insert such an opportunity to HEAP. That is possible
+ only if the node is not needed and there is a parameter for wich all callers
+ pass the same constant. */
+
+static bool
+insert_const_prop_opportunity_to_heap (fibheap_t heap,
+ struct cgraph_node *node,
+ struct ipa_node_params *info)
+{
+ int i;
+ struct cgraph_edge *cs;
+
+ if (node->needed || (i = reevaluate_node_lattices (node, info)) < 0)
+ return false;
+
+ /* Check whether any of the callers and their jump functions can make
+ any of the edges redirectable when updating callgraph because of
+ this particular parameter. If so, bail out. */
+ for (cs = node->callers; cs != NULL; cs = cs->next_caller)
+ {
+ struct ipa_jump_func *jump_func;
+ if (bitmap_bit_p (dead_nodes, cs->caller->uid))
+ continue;
+
+ jump_func = ipa_get_ith_jump_func (IPA_EDGE_REF (cs), i);
+ if (jump_func->type == IPA_JF_PASS_THROUGH
+ && !ipcp_node_is_clone (cs->caller))
+ return false;
+ }
+
+ insert_opportunity_to_heap (heap, INT_MIN, node, i,
+ ipcp_get_lattice (info, i)->constant, 0);
+ return true;
+}
+
+/* Find the best cloning opportunity for NODE and insert into HEAP. */
+
+static void
+insert_best_opportunity_to_heap (fibheap_t heap, struct cgraph_node *node)
+{
+ struct ipa_node_params *info = IPA_NODE_REF (node);
+ int i, chosen_index = -1, parm_count;
+ int caller_count = 0;
+ tree chosen_cst = NULL_TREE;
+ bool got_one = false;
+ struct cgraph_edge *cs;
+ int min_cost = 0, chosen_growth = 0;
+ struct ipcp_lattice *latarr;
+
+
+ gcc_assert (!node->aux);
+ if (!node->analyzed || !node->callers || ipa_is_ipa_cp_ignore (info))
+ return;
+
+ if (dump_file)
+ fprintf (dump_file, "Searching for the best opportunity to clone %s.\n",
+ cgraph_node_name (node));
+
+ /* If possible we always want to do full constant propagation first. */
+ if (insert_const_prop_opportunity_to_heap (heap, node, info)
+ || !optimize_function_for_speed_p (DECL_STRUCT_FUNCTION (node->decl)))
+ return;
+
+ for (cs = node->callers; cs != NULL; cs = cs->next_caller)
+ caller_count++;
+
+ latarr = XALLOCAVEC (struct ipcp_lattice, caller_count);
+ parm_count = ipa_get_param_count (info);
+ for (i = 0; i < parm_count; i++)
+ {
+ int c_idx;
+
+ if (ipcp_parameter_irrelevant_for_cost_p (info, i))
+ continue;
+
+ /* First we'll calculate all incoming lattices for this particular
+ parameter. */
+ for (c_idx = 0, cs = node->callers;
+ cs != NULL;
+ c_idx++, cs = cs->next_caller)
+ {
+ struct ipa_edge_args *args;
+
+ if (bitmap_bit_p (dead_nodes, cs->caller->uid)
+ || ipa_is_ipa_cp_ignore (IPA_NODE_REF (cs->caller)))
+ {
+ latarr[c_idx].type = IPA_BOTTOM;
+ continue;
+ }
+
+ args = IPA_EDGE_REF (cs);
+ ipcp_lattice_from_jfunc (IPA_NODE_REF (cs->caller), &latarr[c_idx],
+ ipa_get_ith_jump_func (args, i));
+ }
+
+ for (c_idx = 0, cs = node->callers;
+ cs != NULL;
+ c_idx++, cs = cs->next_caller)
+ {
+ struct cgraph_edge *e;
+ int cost, growth;
+ int n_calls, freq_sum, e_idx;
+ gcov_type count_sum;
+
+ if (!ipcp_lat_is_const (&latarr[c_idx])
+ || is_string_const (latarr[c_idx].constant))
+ continue;
+
+ count_sum = cs->count + 1;
+ freq_sum = cs->frequency + 2;
+ n_calls = 1;
+ e_idx = c_idx;
+ for (e = cs->next_caller; e != NULL; e = e->next_caller)
+ {
+ e_idx++;
+
+ if (ipcp_lat_is_const (&latarr[e_idx])
+ && ipcp_lats_are_equal (&latarr[c_idx], &latarr[e_idx]))
+ {
+ count_sum += e->count;
+ freq_sum += e->frequency + 1;
+ n_calls++;
+ /* Do not process this one in the outer loop again. */
+ latarr[e_idx].type = IPA_TOP;
+ }
+ }
+
+ cost = calculate_cost_for_cst_arg (info, node, i,
+ latarr[c_idx].constant,
+ n_calls, count_sum, freq_sum,
+ &growth);
+
+ if ((flag_ipa_cp_clone || growth < n_calls)
+ && (!got_one || cost < min_cost))
+ {
+ got_one = true;
+ min_cost = cost;
+ chosen_index = i;
+ chosen_cst = latarr[c_idx].constant;
+ chosen_growth = growth;
+ }
+ }
+ }
+
+ if (got_one)
+ insert_opportunity_to_heap (heap, min_cost, node,
+ chosen_index, chosen_cst, chosen_growth);
+}
+
+/* Return true iff we know that the basic block from which call graph edge CS
+ leads will be removed due to the constant propagation taking place. INFO
+ should describe the caller, CST is the constant that is being propagated to
+ parameter identified by PARM_INDEX. */
+
+static bool
+edge_from_removed_bb_p (struct ipa_node_params *info,
+ int parm_index, tree cst, struct cgraph_edge *cs)
+{
+ int j, b_count = ipa_get_number_of_cond_uses_benefits (info, parm_index);
+ int bb_index = gimple_bb (cs->call_stmt)->index;
+
+ for (j = 0; j < b_count; j++)
+ {
+ struct ipa_param_cond_benefit* ben;
+ bool r;
+
+ ben = ipa_get_parm_cond_benefit (info, parm_index, j);
+ if (integer_nonzerop (fold_build2 (ben->oper, integer_type_node,
+ cst, ben->cst)))
+ r = bitmap_bit_p (ben->true_deleted_bbs, bb_index);
+ else
+ r = bitmap_bit_p (ben->false_deleted_bbs, bb_index);
+
+ if (r)
+ return true;
+ }
+ return false;
+}
+
+/* Execute the cloning described in opportunity OPP and afterwards insert newly
+ recalculated opportunitites into HEAP. */
+
+static void
+seize_cloning_opportunity (fibheap_t heap, struct ipcp_opportunity *opp)
+{
+ struct cgraph_node *node1, *node = opp->node;
+ struct ipa_node_params *info1, *info = IPA_NODE_REF (node);
+ struct cgraph_edge *cs;
+ VEC (cgraph_edge_p, heap) *involved_callers = NULL;
+ VEC (ipa_replace_map_p,gc)* replace_trees = NULL;
+ int parm_count = ipa_get_param_count (info);
+ bitmap args_to_skip;
+ int i, j, k, involved_count = 0;
+ struct ipcp_lattice *latarr;
+ bool mark_dead = true;
+ /* !!! Remove after testing?: */
+ bool got_const = false;
+
+ for (cs = node->callers; cs != NULL; cs = cs->next_caller)
+ {
+ struct ipa_node_params *caller_info = IPA_NODE_REF (cs->caller);
+ struct ipa_edge_args *args;
+ struct ipcp_lattice lat;
+
+ args = IPA_EDGE_REF (cs);
+ ipcp_lattice_from_jfunc (caller_info, &lat,
+ ipa_get_ith_jump_func (args, opp->parm_index));
+ if (ipcp_lat_is_const (&lat)
+ && operand_equal_p (lat.constant, opp->cst, 0))
+ {
+ if (dump_file)
+ fprintf (dump_file, " Will redirect %s -> %s\n",
+ cgraph_node_name (cs->caller),
+ cgraph_node_name (cs->callee));
+
+ VEC_safe_push (cgraph_edge_p, heap, involved_callers, cs);
+ involved_count++;
+ }
+ else
+ mark_dead = false;
+ }
+ gcc_assert (involved_count);
+
+ latarr = XALLOCAVEC (struct ipcp_lattice, parm_count);
+ for (i = 0; i < parm_count; i++)
+ latarr[i].type = IPA_TOP;
+
+ for (j = 0; j < involved_count; j++)
+ {
+ struct ipa_edge_args *args;
+
+ cs = VEC_index (cgraph_edge_p, involved_callers, j);
+ args = IPA_EDGE_REF (cs);
+ for (i = 0; i < parm_count; i++)
+ {
+ struct ipcp_lattice lat;
+
+ ipcp_lattice_from_jfunc (IPA_NODE_REF (cs->caller), &lat,
+ ipa_get_ith_jump_func (args, i));
+ ipa_lattice_meet (&latarr[i], &latarr[i], &lat);
+ }
+ }
+
+ if (dump_file)
+ {
+ fprintf (dump_file, " |");
+ for (i = 0; i < parm_count; i++)
+ {
+ if (latarr[i].type == IPA_BOTTOM)
+ fprintf (dump_file, "B|");
+ else if (latarr[i].type == IPA_TOP)
+ fprintf (dump_file, "T|");
+ else
+ {
+ print_generic_expr (dump_file, latarr[i].constant, 0);
+ fprintf (dump_file, "|");
+ }
+ }
+ fprintf (dump_file, "\n");
+ }
+
+ args_to_skip = BITMAP_GGC_ALLOC ();
+ k = 0;
+ for (i = 0; i < parm_count; i++)
+ {
+ if (ipcp_node_is_clone (node))
+ {
+ /* When cloning clones, the previoulsy-discoverted-to-be-constant and
+ unused arguments are already gone. */
+ if (ipa_is_param_used_for_cloning (info, i)
+ || ipa_is_param_unused (info, i))
+ continue;
+ }
+ else if (ipa_is_param_unused (info, i))
+ {
+ /* We can proactively remove obviously unused arguments. We do this
+ whenever we clone from the original. */
+ bitmap_set_bit (args_to_skip, k);
+ k++;
+ continue;
+ }
+
+ if (ipcp_lat_is_const (&latarr[i]))
+ {
+ struct ipa_replace_map *replace_param;
+ tree parm_tree = ipa_get_param (info, i);
+
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, " Param %i (%i) is cst.\n", i, k);
+ replace_param =
+ ipcp_create_replace_map (parm_tree, &latarr[i]);
+ VEC_safe_push (ipa_replace_map_p, gc, replace_trees, replace_param);
+ bitmap_set_bit (args_to_skip, k);
+ got_const = true;
+ }
+
+ k++;
+ }
+ gcc_assert (got_const);
+
+ node1 = cgraph_create_virtual_clone (node, involved_callers, replace_trees,
+ args_to_skip);
+ info = IPA_NODE_REF (node);
+ info1 = IPA_NODE_REF (node1);
+ gcc_assert (info->params);
+ info1->ipcp_orig_node = info->ipcp_orig_node ? info->ipcp_orig_node : node;
+ if (dump_file)
+ fprintf (dump_file, " New clone name: %s\n",
+ cgraph_node_name (cs->callee));
+
+ for (i = 0; i < parm_count; i++)
+ {
+ if (ipcp_lat_is_const (&latarr[i]))
+ {
+ struct ipcp_lattice *lat = ipcp_get_lattice (info1, i);
+ *lat = latarr[i];
+ ipa_mark_param_used_for_cloning (info1, i);
+ }
+ }
+
+ if (mark_dead)
+ for (cs = node1->callers; cs; cs = cs->next_caller)
+ if (!bitmap_bit_p (dead_nodes, cs->caller->uid)
+ && ipcp_need_redirect_p (cs))
+ {
+ mark_dead = false;
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, " Edge %s -> %s may keep the callee alive.\n",
+ cgraph_node_name (cs->caller),
+ cgraph_node_name (cs->callee));
+ break;
+ }
+ if (mark_dead)
+ {
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, " Marking node dead: %s\n",
+ cgraph_node_name (node));
+ bitmap_set_bit (dead_nodes, node->uid);
+ }
+
+ /* TODO: We can use indirect inlning info to produce new calls. */
+
+
+ for (cs = node->callees; cs; cs = cs->next_callee)
+ if (cs->callee->aux)
+ {
+ struct ipcp_opportunity *co;
+ co = (struct ipcp_opportunity *) fibheap_delete_node (heap,
+ (fibnode_t) cs->callee->aux);
+ cs->callee->aux = NULL;
+ pool_free (opportunity_pool, co);
+ }
+
+ cs = node1->callees;
+ while (cs)
+ {
+ struct cgraph_edge *next_cs = cs->next_callee;
+
+ if (edge_from_removed_bb_p (info1, opp->parm_index, opp->cst, cs))
+ {
+ if (dump_file)
+ fprintf (dump_file, " Edge the new clone to %s deleted "
+ "because it leads from a dead BB.\n",
+ cgraph_node_name (cs->callee));
+ cgraph_remove_edge (cs);
+ }
+ cs = next_cs;
+ }
+
+ /* At this point cgraph callees of node is a superset of calles of node1. */
+ for (cs = node->callees; cs; cs = cs->next_callee)
+ if (!cs->callee->aux)
+ insert_best_opportunity_to_heap (heap, cs->callee);
+
+ if (!mark_dead && !node->aux)
+ insert_best_opportunity_to_heap (heap, node);
+ if (!node1->aux)
+ insert_best_opportunity_to_heap (heap, node1);
+}
+
+/* Make decisions as to what functions should be cloned and what call sites
+ redirected. */
+
+static void
+ipcp_decisions_stage (void)
+{
+ struct cgraph_node *node;
+ fibheap_t heap;
long overall_size = 0, new_size = 0;
long max_new_size;
ipa_check_create_node_params ();
ipa_check_create_edge_args ();
if (dump_file)
- fprintf (dump_file, "\nIPA insert stage:\n\n");
+ fprintf (dump_file, "\nIPA decision making stage:\n");
dead_nodes = BITMAP_ALLOC (NULL);
+ opportunity_pool = create_alloc_pool ("ipa-cp opportunities",
+ sizeof (struct ipcp_opportunity), 128);
for (node = cgraph_nodes; node; node = node->next)
if (node->analyzed)
@@ -1091,139 +1506,52 @@ ipcp_insert_stage (void)
max_new_size = PARAM_VALUE (PARAM_LARGE_UNIT_INSNS);
max_new_size = max_new_size * PARAM_VALUE (PARAM_IPCP_UNIT_GROWTH) / 100 + 1;
- /* First collect all functions we proved to have constant arguments to heap. */
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, " overall_size: %li, max_new_size: %li\n",
+ overall_size, max_new_size);
+
heap = fibheap_new ();
for (node = cgraph_nodes; node; node = node->next)
- {
- struct ipa_node_params *info;
- /* Propagation of the constant is forbidden in certain conditions. */
- if (!node->analyzed || !ipcp_node_modifiable_p (node))
- continue;
- info = IPA_NODE_REF (node);
- if (ipa_is_called_with_var_arguments (info))
- continue;
- if (ipcp_const_param_count (node))
- node->aux = fibheap_insert (heap, ipcp_estimate_cloning_cost (node), node);
- }
+ insert_best_opportunity_to_heap (heap, node);
- /* Now clone in priority order until code size growth limits are met or
- heap is emptied. */
while (!fibheap_empty (heap))
{
- struct ipa_node_params *info;
- int growth = 0;
- bitmap args_to_skip;
- struct cgraph_edge *cs;
+ struct ipcp_opportunity *opp;
+ /* !!! Probably remove the following after I'm done with analysis. */
+ int min_key = (int) fibheap_min_key (heap);
- node = (struct cgraph_node *)fibheap_extract_min (heap);
- node->aux = NULL;
- if (dump_file)
- fprintf (dump_file, "considering function %s\n",
- cgraph_node_name (node));
+ opp = (struct ipcp_opportunity *) fibheap_extract_min (heap);
+ opp->node->aux = NULL;
- growth = ipcp_estimate_growth (node);
-
- if (new_size + growth > max_new_size)
- break;
- if (growth
- && optimize_function_for_size_p (DECL_STRUCT_FUNCTION (node->decl)))
+ if (new_size + opp->growth > max_new_size)
{
- if (dump_file)
- fprintf (dump_file, "Not versioning, cold code would grow");
+ pool_free (opportunity_pool, opp);
+ if (dump_file && (dump_flags & TDF_DETAILS))
+ fprintf (dump_file, "Reached overall growth limit\n");
continue;
}
- new_size += growth;
-
- /* Look if original function becomes dead after clonning. */
- for (cs = node->callers; cs != NULL; cs = cs->next_caller)
- if (cs->caller == node || ipcp_need_redirect_p (cs))
- break;
- if (!cs && !node->needed)
- bitmap_set_bit (dead_nodes, node->uid);
-
- info = IPA_NODE_REF (node);
- count = ipa_get_param_count (info);
-
- replace_trees = VEC_alloc (ipa_replace_map_p, gc, 1);
- args_to_skip = BITMAP_GGC_ALLOC ();
- for (i = 0; i < count; i++)
- {
- struct ipcp_lattice *lat = ipcp_get_lattice (info, i);
- parm_tree = ipa_get_param (info, i);
-
- /* We can proactively remove obviously unused arguments. */
- if (is_gimple_reg (parm_tree)
- && !gimple_default_def (DECL_STRUCT_FUNCTION (node->decl),
- parm_tree))
- {
- bitmap_set_bit (args_to_skip, i);
- continue;
- }
-
- if (lat->type == IPA_CONST_VALUE)
- {
- replace_param =
- ipcp_create_replace_map (parm_tree, lat);
- VEC_safe_push (ipa_replace_map_p, gc, replace_trees, replace_param);
- bitmap_set_bit (args_to_skip, i);
- }
- }
-
- /* Compute how many callers node has. */
- node_callers = 0;
- for (cs = node->callers; cs != NULL; cs = cs->next_caller)
- node_callers++;
- redirect_callers = VEC_alloc (cgraph_edge_p, heap, node_callers);
- for (cs = node->callers; cs != NULL; cs = cs->next_caller)
- VEC_quick_push (cgraph_edge_p, redirect_callers, cs);
-
- /* Redirecting all the callers of the node to the
- new versioned node. */
- node1 =
- cgraph_create_virtual_clone (node, redirect_callers, replace_trees,
- args_to_skip);
- args_to_skip = NULL;
- VEC_free (cgraph_edge_p, heap, redirect_callers);
- replace_trees = NULL;
-
- if (node1 == NULL)
- continue;
if (dump_file)
- fprintf (dump_file, "versioned function %s with growth %i, overall %i\n",
- cgraph_node_name (node), (int)growth, (int)new_size);
- ipcp_init_cloned_node (node, node1);
-
- /* TODO: We can use indirect inlning info to produce new calls. */
+ fprintf (dump_file, "\nCLONING FUNCTION with cost %i: %s\n",
+ min_key, cgraph_node_name (opp->node));
+ new_size += opp->growth;
+ seize_cloning_opportunity (heap, opp);
+ pool_free (opportunity_pool, opp);
if (dump_file)
- dump_function_to_file (node1->decl, dump_file, dump_flags);
-
- for (cs = node->callees; cs; cs = cs->next_callee)
- if (cs->callee->aux)
- {
- fibheap_delete_node (heap, (fibnode_t) cs->callee->aux);
- cs->callee->aux = fibheap_insert (heap,
- ipcp_estimate_cloning_cost (cs->callee),
- cs->callee);
- }
+ fprintf (dump_file, "\n");
}
- while (!fibheap_empty (heap))
- {
- if (dump_file)
- fprintf (dump_file, "skipping function %s\n",
- cgraph_node_name (node));
- node = (struct cgraph_node *) fibheap_extract_min (heap);
- node->aux = NULL;
- }
fibheap_delete (heap);
+ free_alloc_pool (opportunity_pool);
BITMAP_FREE (dead_nodes);
ipcp_update_callgraph ();
ipcp_update_profiling ();
}
+
/* The IPCP driver. */
+
static unsigned int
ipcp_driver (void)
{
@@ -1237,8 +1565,8 @@ ipcp_driver (void)
}
/* 2. Do the interprocedural propagation. */
ipcp_iterate_stage ();
- /* 3. Insert the constants found to the functions. */
- ipcp_insert_stage ();
+ /* 3. Decide what propagation should take place. */
+ ipcp_decisions_stage ();
if (dump_file && (dump_flags & TDF_DETAILS))
{
fprintf (dump_file, "\nProfiling info after insert stage:\n");
@@ -1251,23 +1579,62 @@ ipcp_driver (void)
return 0;
}
-/* Note function body size. */
+/* Perform intraprocedrual analysis needed for ipcp. */
+
+static void
+ipcp_analyze_node (struct cgraph_node *node)
+{
+ /* Unreachable nodes should have been eliminated before ipcp. */
+ gcc_assert (node->needed || node->reachable);
+
+ if (!ipcp_cloneable_function_p (node))
+ ipa_set_ipa_cp_ignore (IPA_NODE_REF (node));
+
+ ipa_initialize_node_params (node);
+ ipa_detect_param_modifications (node);
+ ipa_analyze_params_uses (node);
+}
+
+/* Initialization and computation of IPCP data structures. This is the initial
+ intraprocedural analysis of functions, which gathers information to be
+ propagated later on. */
+
static void
ipcp_generate_summary (void)
{
+ struct cgraph_node *node;
+ struct cgraph_edge *cs;
+
if (dump_file)
- fprintf (dump_file, "\nIPA constant propagation start:\n");
+ fprintf (dump_file, "\nIPA constant propagation summaries creation:\n");
ipa_check_create_node_params ();
ipa_check_create_edge_args ();
ipa_register_cgraph_hooks ();
- /* 1. Call the init stage to initialize
- the ipa_node_params and ipa_edge_args structures. */
- ipcp_init_stage ();
+
+ for (node = cgraph_nodes; node; node = node->next)
+ if (node->analyzed)
+ ipcp_analyze_node (node);
+ for (node = cgraph_nodes; node; node = node->next)
+ {
+ if (!node->analyzed)
+ continue;
+ /* building jump functions */
+ for (cs = node->callees; cs; cs = cs->next_callee)
+ {
+ if (!cs->callee->analyzed)
+ continue;
+ ipa_count_arguments (cs);
+ if (ipa_get_cs_argument_count (IPA_EDGE_REF (cs))
+ != ipa_get_param_count (IPA_NODE_REF (cs->callee)))
+ ipa_set_ipa_cp_ignore (IPA_NODE_REF (cs->callee));
+ ipa_compute_jump_functions (cs);
+ }
+ }
}
/* Gate for IPCP optimization. */
static bool
-cgraph_gate_cp (void)
+gate_ipa_cp (void)
{
return flag_ipa_cp;
}
@@ -1277,7 +1644,7 @@ struct ipa_opt_pass_d pass_ipa_cp =
{
IPA_PASS,
"cp", /* name */
- cgraph_gate_cp, /* gate */
+ gate_ipa_cp, /* gate */
ipcp_driver, /* execute */
NULL, /* sub */
NULL, /* next */
Index: icln/gcc/Makefile.in
===================================================================
--- icln.orig/gcc/Makefile.in 2009-07-31 20:12:47.000000000 +0200
+++ icln/gcc/Makefile.in 2009-07-31 20:12:53.000000000 +0200
@@ -2788,7 +2788,7 @@ ipa-prop.o : ipa-prop.c $(CONFIG_H) $(SY
ipa-cp.o : ipa-cp.c $(CONFIG_H) $(SYSTEM_H) coretypes.h \
$(TREE_H) $(TARGET_H) $(CGRAPH_H) $(IPA_PROP_H) $(TREE_FLOW_H) \
$(TREE_PASS_H) $(FLAGS_H) $(TIMEVAR_H) $(DIAGNOSTIC_H) $(TREE_DUMP_H) \
- $(TREE_INLINE_H) $(FIBHEAP_H) $(PARAMS_H)
+ $(TREE_INLINE_H) $(FIBHEAP_H) $(PARAMS_H) alloc-pool.h
matrix-reorg.o : matrix-reorg.c $(CONFIG_H) $(SYSTEM_H) coretypes.h \
$(TM_H) $(TREE_H) $(RTL_H) $(TREE_INLINE_H) $(TREE_FLOW_H) \
tree-flow-inline.h langhooks.h $(HASHTAB_H) $(TOPLEV_H) $(FLAGS_H) $(GGC_H) \
Index: icln/gcc/ipa-prop.h
===================================================================
--- icln.orig/gcc/ipa-prop.h 2009-07-31 20:12:47.000000000 +0200
+++ icln/gcc/ipa-prop.h 2009-07-31 20:12:53.000000000 +0200
@@ -199,6 +199,10 @@ struct ipa_param_descriptor
unsigned called : 1;
/* Whether the parameter has been used in a condition statement. */
unsigned cond_use : 1;
+ /* Whether this parameter has already been used for cloning and thus should
+ be considered when updating a call graph and not be considered when
+ looking for further opportunities for cloning. */
+ unsigned used_for_cloning : 1;
};
/* ipa_node_params stores information related to formal parameters of functions
@@ -206,10 +210,6 @@ struct ipa_param_descriptor
parameters (such as ipa-cp). */
struct ipa_node_params
{
- /* Number of formal parameters of this function. When set to 0,
- this function's parameters would not be analyzed by the different
- stages of IPA CP. */
- int param_count;
/* Pointer to an array of structures describing individual formal
parameters. */
struct ipa_param_descriptor *params;
@@ -231,9 +231,13 @@ struct ipa_node_params
one. */
gcov_type count_scale;
- /* Whether this function is called with variable number of actual
- arguments. */
- unsigned called_with_var_arguments : 1;
+ /* Number of formal parameters of this function. When set to 0,
+ this function's parameters would not be analyzed by the different
+ stages of IPA CP. */
+ int param_count;
+
+ /* Set when ipa-cp should leave this function alone. */
+ unsigned ipa_cp_ignore : 1;
/* Whether the modification analysis has already been performed. */
unsigned modification_analysis_done : 1;
/* Whether the param uses analysis has already been performed. */
@@ -282,6 +286,23 @@ ipa_is_param_unused (struct ipa_node_par
return info->params[i].unused;
}
+/* Return the used_for_cloning flag corresponding to the Ith formal parameter
+ of the function associated with INFO. */
+
+static inline bool
+ipa_is_param_used_for_cloning (struct ipa_node_params *info, int i)
+{
+ return info->params[i].used_for_cloning;
+}
+
+/* Mark the Ith parameter as one used in cloning. */
+
+static inline void
+ipa_mark_param_used_for_cloning (struct ipa_node_params *info, int i)
+{
+ info->params[i].used_for_cloning = 1;
+}
+
/* Return the modification flag corresponding to the Ith formal parameter of
the function associated with INFO. Note that there is no setter method as
the goal is to set all flags when building the array in
@@ -331,24 +352,23 @@ ipa_get_parm_cond_benefit (struct ipa_no
return &info->params[parm_i].cond_benefits[ben_i];
}
-/* Flag this node as having callers with variable number of arguments. */
+/* Flag this node so that ipa-cp does not modify it. */
static inline void
-ipa_set_called_with_variable_arg (struct ipa_node_params *info)
+ipa_set_ipa_cp_ignore (struct ipa_node_params *info)
{
- info->called_with_var_arguments = 1;
+ info->ipa_cp_ignore = 1;
}
-/* Have we detected this node was called with variable number of arguments? */
+/* Return true iff the function described by info is flagged to be ignored by
+ ipa-cp. */
static inline bool
-ipa_is_called_with_var_arguments (struct ipa_node_params *info)
+ipa_is_ipa_cp_ignore (struct ipa_node_params *info)
{
- return info->called_with_var_arguments;
+ return info->ipa_cp_ignore;
}
-
-
/* ipa_edge_args stores information related to a callsite and particularly
its arguments. It is pointed to by a field in the
callsite's corresponding cgraph_edge. */
Index: icln/gcc/cgraph.c
===================================================================
--- icln.orig/gcc/cgraph.c 2009-07-31 20:01:19.000000000 +0200
+++ icln/gcc/cgraph.c 2009-07-31 20:12:53.000000000 +0200
@@ -1696,7 +1696,7 @@ cgraph_create_virtual_clone (struct cgra
unsigned i;
struct cgraph_edge *e;
- gcc_assert (tree_versionable_function_p (old_decl));
+ gcc_assert (old_node->clone_of || tree_versionable_function_p (old_decl));
/* Make a new FUNCTION_DECL tree node */
if (!args_to_skip)
Index: icln/gcc/cgraphunit.c
===================================================================
--- icln.orig/gcc/cgraphunit.c 2009-07-31 20:01:19.000000000 +0200
+++ icln/gcc/cgraphunit.c 2009-07-31 20:12:54.000000000 +0200
@@ -731,9 +731,18 @@ verify_cgraph_node (struct cgraph_node *
}
else
{
+ /* FIXME: Cloning now discards some edges it knows
+ won't be taken. Unfortunately this is cgraph
+ property is verified only after there is no trace of
+ cloning left so we have to disable this check for
+ all stmts and all functions. At least until we
+ manage not to materialize given BBs. */
+
+ /*
error ("missing callgraph edge for call stmt:");
debug_gimple_stmt (stmt);
error_found = true;
+ */
}
}
}
Index: icln/gcc/tree-inline.c
===================================================================
--- icln.orig/gcc/tree-inline.c 2009-07-31 20:12:47.000000000 +0200
+++ icln/gcc/tree-inline.c 2009-07-31 20:12:54.000000000 +0200
@@ -1334,6 +1334,42 @@ remap_gimple_stmt (gimple stmt, copy_bod
return copy;
}
+/* !!! Doc */
+
+static void
+create_missing_cg_edge (copy_body_data *id, basic_block bb, gimple stmt)
+{
+ tree fndecl = gimple_call_fndecl (stmt);
+ struct cgraph_node *dest;
+
+ if (!fndecl)
+ return;
+
+ dest = cgraph_node (fndecl);
+ /* This can happen if IPA-CP this call is not happening and a remaining one
+ got inlined. Don't create edges or verifier will go berserk. */
+ if (dest->global.inlined_to)
+ return;
+
+ /* We have missing edge in the callgraph. This can happen either when
+ previous inlining turned indirect call into direct call by constant
+ propagating arguments or when IPA-CP knows the whole BB is going away
+ anyway. */
+ if (id->transform_call_graph_edges == CB_CGE_MOVE_CLONES)
+ cgraph_create_edge_including_clones
+ (id->dst_node, dest, stmt, bb->count,
+ compute_call_stmt_bb_frequency (id->dst_node->decl, bb),
+ bb->loop_depth, CIF_ORIGINALLY_INDIRECT_CALL);
+ else
+ cgraph_create_edge (id->dst_node, dest, stmt,
+ bb->count, CGRAPH_FREQ_BASE,
+ bb->loop_depth)->inline_failed
+ = CIF_ORIGINALLY_INDIRECT_CALL;
+ if (dump_file)
+ fprintf (dump_file, "Created new direct edge to %s",
+ cgraph_node_name (dest));
+}
+
/* Copy basic block, scale profile accordingly. Edges will be taken care of
later */
@@ -1406,8 +1442,6 @@ copy_bb (copy_body_data *id, basic_block
need to process all of them. */
do
{
- tree fn;
-
stmt = gsi_stmt (copy_gsi);
if (is_gimple_call (stmt)
&& gimple_call_va_arg_pack_p (stmt)
@@ -1530,33 +1564,8 @@ copy_bb (copy_body_data *id, basic_block
if ((!edge
|| (edge->indirect_call
&& id->transform_call_graph_edges == CB_CGE_MOVE_CLONES))
- && is_gimple_call (stmt)
- && (fn = gimple_call_fndecl (stmt)) != NULL)
- {
- struct cgraph_node *dest = cgraph_node (fn);
-
- /* We have missing edge in the callgraph. This can happen
- when previous inlining turned an indirect call into a
- direct call by constant propagating arguments. In all
- other cases we hit a bug (incorrect node sharing is the
- most common reason for missing edges). */
- gcc_assert (dest->needed || !dest->analyzed);
- if (id->transform_call_graph_edges == CB_CGE_MOVE_CLONES)
- cgraph_create_edge_including_clones
- (id->dst_node, dest, stmt, bb->count,
- compute_call_stmt_bb_frequency (id->dst_node->decl, bb),
- bb->loop_depth, CIF_ORIGINALLY_INDIRECT_CALL);
- else
- cgraph_create_edge (id->dst_node, dest, stmt,
- bb->count, CGRAPH_FREQ_BASE,
- bb->loop_depth)->inline_failed
- = CIF_ORIGINALLY_INDIRECT_CALL;
- if (dump_file)
- {
- fprintf (dump_file, "Created new direct edge to %s",
- cgraph_node_name (dest));
- }
- }
+ && is_gimple_call (stmt))
+ create_missing_cg_edge (id, bb, stmt);
flags = gimple_call_flags (stmt);
if (flags & ECF_MAY_BE_ALLOCA)
@@ -3317,6 +3326,11 @@ expand_call_inline (basic_block bb, gimp
cg_edge = cgraph_edge (id->dst_node, stmt);
+ /* IPA-CP can remove edges from the cgraph if it knows they won't be taken.
+ If this is the case, bail out. */
+ if (!cg_edge)
+ goto egress;
+
/* Don't try to inline functions that are not well-suited to
inlining. */
if (!cgraph_inline_p (cg_edge, &reason))