Fwd: LLVM collaboration?

[Uday Khedker]

On Tuesday 11 February 2014 09:30 PM, Jan Hubicka wrote:
>>
>>
>>
>>
>> On Tuesday 11 February 2014 03:25 PM, Renato Golin wrote:
>>> Hi Jan,
>>>
>>> I think this is a very good example where we could all collaborate
>>> (including binutils).
>>>
>>> I'll leave your reply intact, so that Chandler (CC'd) can get a bit
>>> more context. I'm copying him because he (and I believe Diego) had
>>> more contact with LTO than I had.
>>>
>>> If I got it right, LTO today:
>>>
>>> - needs the drivers to explicitly declare the plugin
>
> Yes.
>>> - needs the library available somewhere
>>> - may have to change the library loading semantics (via LD_PRELOAD)
>
> Not in binutils implementation (I believe it is the case for darwin's libLTO).
> With binutils you only need to pass explicit --plugin argument into all
> tools that care (ld/ar/nm/ranlib)
>>
>> There is another need that I have felt in LTO for quite some time.
>> Currently, it has a non-partitioned mode or a partitioned mode but
>> this decision is taken before the compilation begins. It would be
>> nice to have a mode that allows dynamic loading of function bodies
>> so that a flow and context sensitive IPA could load functions bodies
>> on demand, and unload them when they are not needed.
>
> I implemented on-demand loading of function bodies into GCC-4.8 if I recall
> correctly. Currently I thinko only Martin Liska's code unification pass uses it
> to verify that two function bdoes it thinks are equivalent are actually
> equivalent. Hopefully it will be merged into 4.10.

Great. We will experiment with it.

Uday.

>>
>> Uday.
>>
>>>
>>> Since both toolchains do the magic, binutils has no incentive to
>>> create any automatic detection of objects.
>>>
>>> The part that I didn't get is when you say about backward
>>> compatibility. Would LTO work on a newer binutils with the liblto but
>>> on an older compiler that knew nothing about LTO?
>>>
>>> Your proposal is, then, to get binutils:
>>>
>>> - recognizing LTO logic in the objects
>>> - automatically loading liblto if recognized
>>> - warning if not
>
> I basically think that binutils should have a way for installed compiler to
> register a plugin and load all plugins by default (or perhaps for performance
> or upon detecking an compatible LTO object file in some way, perhaps also by
> information given in the config file) and let them claim the LTO objects they
> understand to.
>
> With the backward compatibility I mean that if we release a new version of
> compiler that can no longer read the LTO objects of older compiler, one can
> just install both versions and have their plugins to claim only LTO objects
> they understand. Just if they were two different compilers.
>
> Finally I think we can make binutils to recognize GCC/LLVM LTO objects
> as a special case and produce friendly message when user try to handle
> them witout plugin as oposed to today strange errors about file formats
> or missing symbols.
>
> Honza
>>>
>>> I'm assuming the extra symbols would be discarded if no library is
>>> found, together with the warning, right? Maybe an error if -Wall or
>>> whatever.
>>>
>>> Can we get someone from the binutils community to opine on that?
>>>
>>> cheers,
>>> --renato
>>>
>>> On 11 February 2014 02:29, Jan Hubicka <hubicka@ucw.cz> wrote:
>>>> One practical experience I have with LLVM developers is sharing experiences
>>>> about getting Firefox to work with LTO with Rafael Espindola and I think it was
>>>> useful for both of us. I am definitly open to more discussion.
>>>>
>>>> Lets try a specific topic that is on my TODO list for some time.
>>>>
>>>> I would like to make it possible for mutliple compilers to be used to LTO a
>>>> single binary. As we are all making LTO more useful, I think it is matter of
>>>> time until people will start shipping LTO object files by default and users
>>>> will end up feeding them into different compilers or incompatible version of
>>>> the same compiler. We probably want to make this work, even thought the
>>>> cross-module optimization will not happen in this case.
>>>>
>>>> The plugin interface in binutils seems to do its job well both for GCC and LLVM
>>>> and I hope that open64 and ICC will eventually join, too.
>>>>
>>>> The trouble however is that one needs to pass explicit --plugin argument
>>>> specifying the particular plugin to load and so GCC ships with its own wrappers
>>>> (gcc-nm/gcc-ld/gcc-ar and the gcc driver itself) while LLVM does similar thing.
>>>>
>>>> It may be smoother if binutils was able to load multiple plugins at once and
>>>> grab plugins from system and user installed compilers without explicit --plugin
>>>> argument.
>>>>
>>>> Binutils probably should also have a way to detect LTO object files and produce
>>>> more useful diagnostic than they do now, when there is no plugin claiming them.
>>>>
>>>> There are some PRs filled on the topic
>>>> http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=15300
>>>> http://cygwin.com/frysk/bugzilla/show_bug.cgi?id=13227
>>>> but not much progress on them.
>>>>
>>>> I wonder if we can get this designed and implemented.
>>>>
>>>> On the other hand, GCC current maintains non-plugin path for LTO that is now
>>>> only used by darwin port due to lack of plugin enabled LD there.  It seems
>>>> that liblto used by darwin is losely compatible with the plugin API, but it makes
>>>> it harder to have different compilers share it (one has to LD_PRELOAD liblto
>>>> to different one prior executing the linker?)
>>>>
>>>> I wonder, is there chance to implement linker plugin API to libLTO glue or add
>>>> plugin support to native Darwin tools?
>>>>
>>>> Honza