Re: [whopr] Design/implementation alternatives for the driver and WPA

[Chris Lattner <clattner at apple dot com>]

On Jun 4, 2008, at 7:22 AM, Ian Lance Taylor wrote:
> Chris Lattner <clattner@apple.com> writes:
> > Is there a specific reason you don't use the LLVM LTO interface?  It
> > seems to be roughly the same as your proposed interface:
> >
> > a) it has a simple C interface like your proposed one
> > b) it is already implemented in one system linker (Apple's), so GCC
> > would just provide its own linker plugin and it would work on apple
> > platforms
> > c) it is richer than your interface
> > d) it is battle tested, and exists today
> > e) it is completely independent of llvm (by design)
> > f) it is fully documented: http://llvm.org/docs/LinkTimeOptimization.html
> >
> > Is there something specific you don't like about the LLVM interface?
>
> (I didn't design the proposed linker interface, and I'm not sure my
> earlier comments were included in the proposal sent to the list.  I'm
> going to reply to that next.)
>
> When I look at the LLVM interface as described on that web page, I see
> these issues, all fixable:
> * No support for symbol versioning.

Very true.  I think it would great to work from a common model that  
can be extended to support both compilers.  Having a unified interface  
would be very useful, and we are happy to evolve the interface to suit  
more general needs.

> * The return value of lto_module_get_symbol_attributes is not
>  defined.

Ah, sorry about that.  Most of the details are actually in the public  
header.  The result of this function is a 'lto_symbol_attributes'  
bitmask.  This should be more useful and revealing:
http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm-c/lto.h?revision=HEAD&view=markup

> * lto_codegen_set_debug_model and lto_codegen_set_pic_model appear to
>  be underspecified--don't they need an additional parameter?

These are actually likely to change.  We are currently working on  
extending the model to better handle the case whentranslation units  
are compiled with different flags.  I expect this to subsume the debug  
and pic handling, which are pretty ad-hoc right now.  There should be  
a proposal going out to llvmdev in the next few days on this.

> * Interfaces like lto_module_get_symbol_name and
>  lto_codegen_add_must_preserve_symbol are inefficient when dealing
>  with large symbol tables.

The intended model is for the linker to query the LTO plugin for its  
symbol list and build up its own linker-specific hash table.  This way  
you don't need to force the linker to use the plugin's data structure  
or the plugin to use the linker data structure.  We converged on this  
approach after trying it the other way.

Does this make sense, do you have a better idea?

> A more general problem is that the main reason I see to use a linker
> plugin is to let the linker handle symbol resolution.

There is that, but also it lets the linker handle things like export  
maps, visibility, strange platform-specific options, etc.  As you  
know, linker's are very complex :)

> The LLVM
> interface does not do that.

Yes it does, the linker fully handles symbol resolution in our model.

> Suppose the linker is invoked on a
> sequence of object files, some with with LTO information, some
> without, all interspersed.  Suppose some symbols are defined in
> multiple .o files, through the use of common symbols, weak symbols,
> and/or section groups.  The LLVM interface simply passes each object
> file to the plugin.

No, the native linker handles all the native .o files.

>  The result is that the plugin is required to do
> symbol resolution itself.  This 1) loses one of the benefits of having
> the linker around; 2) will yield incorrect results when some non-LTO
> object is linked in between LTO objects but redefines some earlier
> weak symbol.

In the LLVM LTO model, the plugin only needs to know about its .o  
files, and the linker uses this information to reason about symbol  
merging etc.  The Mac OS X linker can even do dead code stripping  
across Macho .o files and LLVM .bc files.

Further other pieces of the toolchain (nm, ar, etc) also use the same  
interface so that they can return useful information about LLVM LTO  
files.

> Also, returning a single object file restricts the possibilities.  The
> design of WHOPR, as I understand it, permits creating several
> different object files in parallel based on a fast analysis of which
> code should be compiled together.  When the linker supports concurrent
> linking, it will be desirable to be able to provide it with each
> object file as it is completed.

This sounds like a natural and easy extension once whopr gets farther  
along.

This is our second major revision of the LTO interfaces, and the  
interface continues to slowly evolve.  I think it would be great to  
work with you guys to extend the design to support GCC's needs.

-Chris