LTO, shared libraries and extern class templates

[Hadrien Grasland]

Hi Jonathan,

Thanks for clarifying that explicit class template instantiations should 
work. Trying to produce a minimal reproducer was a very interesting 
exercise. As it turns out, the problematic situation which I am 
experiencing does not occur with explicit template instantiations per 
se, but only when those are combined with inheritance in a certain way.

Let us consider a toy library composed of this header...

----

#pragma once

template <typename T>
class Superclass {
   public:
     void print(const T& value) const;
};

template <typename T>
class Subclass : public Superclass<T> {
   public:
     void print_sub(const T& value) const;
};

extern template class Subclass<int>;

----

...and this source file...

----

#include <iostream>
#include "lib.hpp"

template <typename T>
void Superclass<T>::print(const T& value) const {
     std::cout << "The value is " << value << std::endl;
}

template <typename T>
void Subclass<T>::print_sub(const T& value) const {
     this->print(value);
}

template class Subclass<int>;

----

On a regular build of this library, both Superclass<int>::print() and 
Subclass<int>::print_sub() are exposed as weak symbols and can be linked 
against:

----

00000000000011da  w    F .text  0000000000000051              Superclass<int>::print(int const&) const
00000000000011b4  w    F .text  0000000000000026              Subclass<int>::print_sub(int const&) const

----

On an LTO build from GCC 8.2, however, Superclass<int>::print() becomes 
a local symbol which cannot be linked against:

----

000000000000116c l     F .text  0000000000000050              Superclass<int>::print(int const&) const
0000000000001146 g     F .text  0000000000000025              Subclass<int>::print_sub(int const&) const

----

If Superclass<int> is also explicitly instantiated, both symbols will be 
global in the LTO build too. However, having to explicitly instantiate 
the whole class hierarchy like this could quickly get tedious.

I would spontaneously expect an explicit instantiation of Subclass to 
feature all the code that one needs in order to use Subclass, including 
the Superclass code. Is the behavior above expected ?

Cheers,
Hadrien


Le 11/10/2018 à 15:44, Jonathan Wakely a écrit :
> On Thu, 11 Oct 2018 at 14:01, Hadrien Grasland <grasland@lal.in2p3.fr> wrote:
>> Hi everyone,
>>
>> This is a follow-up to my previous e-mail about linkers, now that I
>> understand the problem that I'm dealing with well enough to describe it :)
>>
>> I'm trying to enable LTO in a large, heavily templated C++ codebase,
>> whose developers made a lot of effort to improve compilation efficiency.
>> To this end...
>>
>>    * Significant effort is expended in separating class declarations and
>>      definitions in regular OO code
>>    * C++11's "extern template" and explicit template instantiations are
>>      extensively used to reduce template-induced duplicate work.
>>    * Shared libraries are also used, sometimes in a cascading fashion
>>      where an executable depends on library A, which depends on another
>>      library B
>>
>> When I enable LTO, I notice that many weak function template symbols
>> turn into local symbols, and are thus not available anymore to clients
>> of a shared library. This is a somewhat expected side-effect of LTO,
>> since AFAIK one of its design goals is to discard unused symbols.
>>
>> My problem is that this weak symbol pruning process is in my case a
>> little bit too agressive. For example, in the "cascading" scenario
>> above, I end up in a situation where library A fails to link because it
>> cannot find the function symbols associated with "extern template class"
>> declarations in the headers of library B. Note that this only happens
>> with extern template classes: extern template functions keep working as
>> expected.
> That suggests a bug in library B. The explicit instantiation
> declaration (the "extern template" bit) needs to be matched by an
> explicit instantiation *definition*. That definition should not be
> weak, and so should not be discarded.
>
> It sounds like you've been using "extern template" to suppress
> implicit instantiations, and then assuming that some other library
> will happen to provide the definition via some other implicit
> instantiation. With LTO the other implicit instantiations are being
> pruned, and the assumption fails.
>
> If you're doing it properly, using explicit instantiation definitions
> to provide the needed symbols, then they should not get dropped by the
> LTO linker. If they do, I think that's a bug in GCC and/or the linker.
>
>> I can work around this by providing library A with the definitions of
>> the relevant templates instead of the declarations, falling back to a
>> classic duplicate instantiation model, but I am curious if there is a
>> better way. Can I somehow structure or annotate the C++ code so that the
>> linker knows that the explicit template instantiations of library B are
>> used by its clients (like library A) and must be kept around as weak or
>> global symbols of the output shared library?
> Does library B really have explicit instantiation definitions? When I
> create a shared library using LTO any explicit instantiations are
> turned into GLOBAL symbols, not WEAK, and are not removed.
>
> It would help if you can create a minimal example showing the problem
> you describe.