[Bug c++/89087] New: Dllexport for explicit template instantiation with nested classes loses nested class

[martin at martin dot st]

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=89087

            Bug ID: 89087
           Summary: Dllexport for explicit template instantiation with
                    nested classes loses nested class
           Product: gcc
           Version: 8.2.0
            Status: UNCONFIRMED
          Severity: normal
          Priority: P3
         Component: c++
          Assignee: unassigned at gcc dot gnu.org
          Reporter: martin at martin dot st
  Target Milestone: ---

Created attachment 45537
  --> https://gcc.gnu.org/bugzilla/attachment.cgi?id=45537&action=edit
Sample code showing the issue

When an explicit template instantiation of a template class with a nested class
is declared with the dllexport attribute, only members from the outer class
actually gets the embedded export directive.

A caller that sees the explicit template instantiation declaration won't emit
those symbols but produce undefined references to them (both for the outer and
inner class), relying on the template instantiation in a different translation
unit.

If relying on the dllexport attribute for exporting the relevant symbols, only
the outer class' members are exported, and linking to the dll fais.


To showcase the problem:

header.h:
template <class T> struct outer {
  void f();
  struct inner {
    void f();
  };
};

template <class T> void outer<T>::f() {}
template <class T> void outer<T>::inner::f() {}

extern template class
#ifdef DLLEXPORT
__declspec(dllexport)
#elif defined(DLLIMPORT)
__declspec(dllimport)
#endif
outer<char>;


lib.cpp:
#define DLLEXPORT
#include "header.h"

template class outer<char>;

caller.cpp:
#define DLLIMPORT
#include "header.h"

int main(int argc, char* argv[]) {
  outer<char> a;
  a.f();
  outer<char>::inner b;
  b.f();
  return 0;
}


Building this fails in this way:
$ make
x86_64-w64-mingw32-g++    -c -o caller.o caller.cpp
x86_64-w64-mingw32-g++    -c -o lib.o lib.cpp
x86_64-w64-mingw32-g++ -shared -o lib.dll lib.o -Wl,--out-implib,liblib.dll.a
x86_64-w64-mingw32-g++ -o caller.exe caller.o -L. -llib
caller.o:caller.cpp:(.text+0x28): undefined reference to
`outer<char>::inner::f()'
collect2: error: ld returned 1 exit status
Makefile:5: recipe for target 'caller.exe' failed
make: *** [caller.exe] Error 1


The template instantiation in lib.cpp does get both outer and inner function
definitions:
$ x86_64-w64-mingw32-nm lib.o
<snip>
0000000000000000 T _ZN5outerIcE1fEv
0000000000000000 T _ZN5outerIcE5inner1fEv

And the caller gets undefined references to the same:
$ x86_64-w64-mingw32-nm caller.o 
<snip>
0000000000000000 T main
                 U _ZN5outerIcE1fEv
                 U _ZN5outerIcE5inner1fEv

But only the outer function actually ended up exported from the DLL:
$ x86_64-w64-mingw32-objdump -s lib.o
<snip>
Contents of section .drectve:
 0000 202d6578 706f7274 3a225f5a 4e356f75   -export:"_ZN5ou
 0010 74657249 63453166 45762200           terIcE1fEv".    


If the DLL is linked with -Wl,--export-all-symbols, both functions are exported
from the DLL and linking succeeds.


This is contrary to MSVC (which admittedly has got an entirely different C++
ABI). In MSVC, the caller emits the inner class' methods despite the explicit
template instantiation (both when the template instantiation was marked
dllimport, but also if dllimport is omitted):

With dllimport:
$ cl -nologo -c caller.cpp 
caller.cpp
$ x86_64-w64-mingw32-nm caller.obj 
<snip>
0000000000000000 T ?f@inner@?$outer@D@@QEAAXXZ
                 U __imp_?f@?$outer@D@@QEAAXXZ
0000000000000000 T main

Without dllimport:
$ cat caller.cpp | sed 's/^#def.*//' > caller-nodllimport.cpp
$ ~/msvc2017/bin64/cl -nologo -c caller-nodllimport.cpp 
caller-nodllimport.cpp
$ x86_64-w64-mingw32-nm caller-nodllimport.obj 
<snip>
0000000000000000 T ?f@inner@?$outer@D@@QEAAXXZ
                 U ?f@?$outer@D@@QEAAXXZ
0000000000000000 T main



To solve this (short of requiring using -Wl,--export-all-symbols on any library
that uses explicit template instantiation with nested classes), the dllexport
either needs to cover the nested class, or an explicit template instantiation
should only be considered to cover the outer class.