Re: Inlining, nested functions, and VLAs

[Jan Hubicka <hubicka at ucw dot cz>]

> This comes in with an ICE for the ACATS test c61008a.  This was complex and
> took quite a while to understand, but here's the gist of the situation:
> 
> Suppose we have a function F with a nested function Q.  Q has one argument,
> which we'll call P, of type T.  But T depends on some variables in F and F
> contains a call to Q, with, say, value V.
> 
> Now F gets inlined and let's look at handling of the Q (V) call.  V has type
> T, but T depends on variables in F and so *does* get remapped.  However, we
> haven't remapped the PARM_DECL in Q, so we are making a call to Q with
> mismatched types.  This causes a subsequent ICE in fold_convert when we try
> to inline Q.
> 
> It seems to me that the right way to inline this is to make a copy of Q
> remapping it's argument types with respect to F, but I don't think that makes
> sense.
> 
> So I think the alternative is not to inline a function if it has nested
> functions whose argument types depend on variables in that function.  Does
> that seem right?  (Note that I think we have to do this recursively because Q
> might have normal args but have a nested function Z with an arg that depended
> on variables in F.)  If we do this, what's the best way to look for the
> nested functions, using cgraph?

Before unnesting there is node->origin pointer pointing to the parent
function.  After function unnesting (happening as part of
cgraph_finalize*) the origin pointers are cleaned up so the nodes can be
dealt with independently.  You probably want to simply set flag during
doing the unnesting so the cannot_inline fails.

Honza