LTO, shared libraries and extern class templates

[Jonathan Wakely]

On Thu, 11 Oct 2018 at 14:01, Hadrien Grasland <grasland@lal.in2p3.fr> wrote:
>
> Hi everyone,
>
> This is a follow-up to my previous e-mail about linkers, now that I
> understand the problem that I'm dealing with well enough to describe it :)
>
> I'm trying to enable LTO in a large, heavily templated C++ codebase,
> whose developers made a lot of effort to improve compilation efficiency.
> To this end...
>
>   * Significant effort is expended in separating class declarations and
>     definitions in regular OO code
>   * C++11's "extern template" and explicit template instantiations are
>     extensively used to reduce template-induced duplicate work.
>   * Shared libraries are also used, sometimes in a cascading fashion
>     where an executable depends on library A, which depends on another
>     library B
>
> When I enable LTO, I notice that many weak function template symbols
> turn into local symbols, and are thus not available anymore to clients
> of a shared library. This is a somewhat expected side-effect of LTO,
> since AFAIK one of its design goals is to discard unused symbols.
>
> My problem is that this weak symbol pruning process is in my case a
> little bit too agressive. For example, in the "cascading" scenario
> above, I end up in a situation where library A fails to link because it
> cannot find the function symbols associated with "extern template class"
> declarations in the headers of library B. Note that this only happens
> with extern template classes: extern template functions keep working as
> expected.

That suggests a bug in library B. The explicit instantiation
declaration (the "extern template" bit) needs to be matched by an
explicit instantiation *definition*. That definition should not be
weak, and so should not be discarded.

It sounds like you've been using "extern template" to suppress
implicit instantiations, and then assuming that some other library
will happen to provide the definition via some other implicit
instantiation. With LTO the other implicit instantiations are being
pruned, and the assumption fails.

If you're doing it properly, using explicit instantiation definitions
to provide the needed symbols, then they should not get dropped by the
LTO linker. If they do, I think that's a bug in GCC and/or the linker.

> I can work around this by providing library A with the definitions of
> the relevant templates instead of the declarations, falling back to a
> classic duplicate instantiation model, but I am curious if there is a
> better way. Can I somehow structure or annotate the C++ code so that the
> linker knows that the explicit template instantiations of library B are
> used by its clients (like library A) and must be kept around as weak or
> global symbols of the output shared library?

Does library B really have explicit instantiation definitions? When I
create a shared library using LTO any explicit instantiations are
turned into GLOBAL symbols, not WEAK, and are not removed.

It would help if you can create a minimal example showing the problem
you describe.