[PATCH][RFC] Re-organize how we stream trees in LTO

[Diego Novillo]

On 2012-10-16 10:43 , Richard Biener wrote:
>
> This patch shows work-in-progress (read: implementation uglyness
> hopefully to vanish ...) regarding to moving LTO type merging
> work from WPA to compile stage.

You mean to LTRANS, the stage after WPA, right?

> The patch re-organizes lto_output_tree (the write_tree streamer
> hook for LTO) in a way so that we output all tree fields
> in easy to discover places.  This means that we have forward
> references to trees not yet (fully) written, something the
> current state avoids by "inline-expanding" forward referenced
> trees at the point of reference (which results in interweaved
> trees on disk).  To be able to achieve this lto_output_tree
> now performs a DFS walk along tree edges to collect trees
> that need to be output and outputs them in SCC chunks
> in a new LTO stream container, LTO_tree_scc.

Nice.  The interleaved output is ugly.

> This will allow the reader side at WPA stage to call uniquify
> nodes on each tree SCC, avoiding completely the DFS walks done
> there (hopefully, we do DFS walks for both hashing and comparing
> there - note that WPA gathers type SCCs, not tree SCCs - a subtle
> difference that remains to be seen on whether it is important ...)
>
> One complication is how we handle streaming INTEGER_CSTs.
> When an INTEGER_CST refers to an already input type we can
> allocate it using the build_int_cst_wide routine which takes
> care of putting it into the appropriate hashtables for caching.
> If an INTEGER_CST is part of a SCC (TYPE_DOMAIN values are
> the most obvious cases) we now stream them as regular trees
> (they cannot already exist in any cache as the type is being
> read in) - but the patch does not yet populate the caches
> in case the type ends up prevailing (thus it will produce
> some unshared INTEGER_CSTs for now).
>
> One uglyness of the patch is how I mis-use the streamer hooks
> to switch the tree streamer routines from actually writing
> tree references to performing the DFS walk.  For the DFS walk
> I need information on the edge, thus a 'from' tree argument
> is added to the write_tree hook.  Eventually my plan was
> to transform this to a walk_tree-like interface.
>
> Diego - is PTH still live?  Thus, do I need to bother about
> inventing things in a way that can be hook-ized?

We will eventually revive PPH.  But not in the short term.  I think it 
will come back when/if we start implementing C++ modules.  Jason, 
Lawrence, is that something that you see coming for the next standard?

I suspect that the front end will need to distance itself from 'tree' 
and have its own streamable IL.  So, the hooks may not be something we 
need to keep long term.

Emitting the trees in SCC groups should not affect the C++ streamer too 
much.  It already is doing its own strategy of emitting tree headers so 
it can do declaration and type merging.  As long as the trees can be 
fully materialized from the SCC groups, it should be fine.

> forsee any complication for PTH or C++ modules the way
> I re-arranged things?  Any good (C++) ideas on how to
> design this lto_walk_tree?

Sorry.  What lto_walk_tree?

> --- 44,50 ----
>         tree itself.  The second boolean parameter specifies this for
>         the tree itself, the first for all siblings that are streamed.
>         The referencing mechanism is up to each streamer to implement.  */
> !   void (*write_tree) (struct output_block *, tree, tree, bool, bool);

You want to document what the new 'tree' argument does here.

> ! #define stream_write_tree_edge_shallow_non_ref(OB, FROM, TO, REF_P) \
> !     streamer_hooks.write_tree(OB, FROM, TO, REF_P, false)

Why the stream_write_tree_edge() naming?  Not sure what you mean by this.


Diego.