Linker API Proposal

["Kirkegaard, Knud J" <knud dot j dot kirkegaard at intel dot com>]

We understand that gcc is considering adding link-time compilation
features and an API to allow the linker to interact with the compiler
and related tools.  Here are some thoughts on what sorts of
interactions compilers and other program development tools might have
with the linker through such an API.

We welcome any comments. We would like to achieve a complete linker API
proposal.


Tools that Might Use the Proposed API
-------------------------------------

These tools might benefit from an API to the linker (ld) and archiver
(ar):

o Compilers

o Binary instrumentation tools
  - profilers
  - profile-directed data layout and function ordering

o Performance analysis tools

o Static analysis tools
  - symbol cross-indexer


Compiler/Linker Interface
-------------------------

Other compilers, besides gcc, could benefit from a linker API.
When performing a whole program interprocedural optimization (IPO)
compilation, Intel's compiler currently performs two passes.  In the
first pass, it generates a "container" object file for each source file
on the command line.  The container file holds the intermediate
representation (IR) data for the compilation rather than the object
code.  In the second pass, the compiler "plays linker".  It reads all of
the object files and performs all of the library searches that the
linker would have.  The compiler along the way collects symbol table
information and reads back any IR it encounters.  It then compiles the
IR and writes the object code to conventional object files.  The driver
then invokes the linker to build a program image from those object
files.

A linker API would relieve the need of the compiler to "play linker".
No longer would it have to understand linker symbol search semantics
and to track changes in linker behavior from release to release.  The
information that the compiler collects today during the second pass of
an IPO compilation is:

o object file symbol table information:
  - scope (local vs. global)
  - visibility (preemptable/protected/hidden/internal)
  - referenced but defined elsewhere (needed for local to compilation
    unit analysis)
o object file section header information (so that for example the
  compiler can compute the total size of the small data area)
o IR from files compiled in pass 1 or archived from a previous IP
  compilation

The control and data flow that must pass over the API for an IPO
compilation is thus:

1) As the linker processes object files (including those retrieved
while searching archives and shared objects to resolve undefined
symbols), the linker passes the object to the compiler.  The
compiler checks the file for IR and retrieves the symbol table
information.  If the object is an IR container file, the compiler
tells the linker not to process the file any further, and gives the
linker a list of symbols to be removed from its undefined symbol list
because they are resolved by the IR.

2) When the linker finishes its initial pass over the object files, it
must call the compiler, which then performs the IPO compilation of the
IR that it has collected.  The result is one or more new object files
that the linker must process as it has all previous object files.

3) The rest of the linking operation proceeds as it would in the
non-IPO case.


Compiler/Archiver Interface
---------------------------

Intel currently ships its own archiver tool (xiar) as a wrapper around
ar that invokes the compiler if a container object file containing IR
is being archived.  We could eliminate this extra tool if the archiver
had an interface to invoke the compiler to generate a full object file
when an IR container object module is added to the archive.


Symbol Cross-indexer
--------------------

This is a new tool that becomes possible if there is a linker API.
The tool would record linker symbol table activity and would produce a
cross-reference index showing each link-time symbol, the file that
defined it, the symbol's final value, and the files that referenced it
(i.e., had it as "undefined" in their symbol tables).


Miscellaneous API Requirements
------------------------------

1) Versioning and upward compatibility.  There must be a method
(major/minor identification field; API feature bit vector) whereby
the linker and tools can detect that they speak incompatible versions
of the API.  Effort should be made to keep changes to the API upward
compatible.

2) The API should support participation by more than one compilation
system in the link.  For example, a link might involve some files
containing Intel IR and some containing IR from a GNU compiler.  The
API must specify a usage protocol for handling this situation.