Obscure inline/nesting interaction (ACATS c61008a)

[Richard Kenner]

The two of us discussed this a while back and I think this is what we
came up with.  The comments I added explain the issue.

This fixes ACATS tests c61008a.

I'm starting testing now in x86_64-linux-gnu.

As I imply in the comments, I'm not thrilled with this solution, but don't
see a better one.  Do you?

2004-11-28  Richard Kenner  <kenner@vlsi1.ultra.nyu.edu>

	* tree-nested.c (check_for_nested_with_variably_modified): New.
	(create_nesting_tree): Call it.

*** tree-nested.c	6 Oct 2004 23:45:48 -0000	2.20
--- tree-nested.c	28 Nov 2004 14:45:12 -0000
*************** walk_all_functions (walk_tree_fn callbac
*** 613,618 ****
    while (root);
  }
- 
  
  /* Construct our local datastructure describing the function nesting
     tree rooted by CGN.  */
--- 613,655 ----
    while (root);
  }
  
+ /* We have to check for a fairly pathalogical case.  The operands of function
+    nested function are to be interpreted in the context of the enclosing
+    function.  So if any are variably-sized, they will get remapped when the
+    enclosing function is inlined.  But that remapping would also have to be
+    done in the types of the PARM_DECLs of the nested function, meaning we'd
+    have to make a copy of the nested function corresponding to each time the
+    enclosing function was inlined.  That's not practical, so we don't inline
+    such an enclosing function.
+ 
+    We have to do that check recursively, so here return indicating whether
+    FNDECL has such a nested function.  ORIG_FN is the function we were
+    trying to inline to use for checking whether any argument is variably
+    modified by anything in it.
+ 
+    It would be better to do this in tree-inline.c so that we could give
+    the appropriate warning for why a function can't be inlined, but that's
+    too late since the nesting structure has already been flattened and
+    adding a flag just to record this fact seems a waste of a flag.  */
+ 
+ static bool
+ check_for_nested_with_variably_modified (tree fndecl, tree orig_fndecl)
+ {
+   struct cgraph_node *cgn = cgraph_node (fndecl);
+   tree arg;
+ 
+   for (cgn = cgn->nested; cgn ; cgn = cgn->next_nested)
+     {
+       for (arg = DECL_ARGUMENTS (cgn->decl); arg; arg = TREE_CHAIN (arg))
+ 	if (variably_modified_type_p (TREE_TYPE (arg), 0), orig_fndecl)
+ 	  return true;
+ 
+       if (check_for_nested_with_variably_modified (cgn->decl, orig_fndecl))
+ 	return true;
+     }
+ 
+   return false;
+ }
+ 
  /* Construct our local datastructure describing the function nesting
     tree rooted by CGN.  */
*************** create_nesting_tree (struct cgraph_node 
*** 633,636 ****
--- 670,678 ----
      }
  
+   /* See discussion at check_for_nested_with_variably_modified for a
+      discussion of why this has to be here.  */
+   if (check_for_nested_with_variably_modified (info->context, info->context))
+     DECL_UNINLINABLE (info->context) = true;
+ 
    return info;
  }