Linking Patterns and Tricks

Introduction

Binary compatibility on Linux has long been a root cause of many problems. As a user, I absolutely hate seeing errors like version GLIBC_2.11 not found pop up when trying to run a prebuilt binary. In fact, some software takes this into account when releasing, and tries—through certain build techniques, especially linking tricks—to make the resulting binaries work across various Linux distributions, i.e., cross-distribution compatible. For our internal software products, we obviously should aim to support as many distributions as possible. This article mainly summarizes the related techniques.

Portable Binary

Executable

For executables, gcc supports the -static option. It ensures that the produced executable can completely disable dynamic linking, so the linked executable contains no references to any shared libraries. But then you need to ensure the compiler can find static artifacts like libc.a so it can link correctly; otherwise you will see errors like "-lc" not found. The benefit of such executables is that they can (possibly) run correctly on any Linux machine without triggering missing-dependency errors. Therefore, some tool-like programs are well suited to be linked this way.

Shared Library

For a compiled .so, the following libraries are generally inconvenient—or unnecessary—to link statically:

linux-vdso.so.1 =>  (0x00007ffff53ff000)
librt.so.1 => /lib64/librt.so.1 (0x00007ffee0209000)
libm.so.6 => /lib64/libm.so.6 (0x00007ffedff84000)
libc.so.6 => /lib64/libc.so.6 (0x00007ffedfbf0000)
/lib64/ld-linux-x86-64.so.2 (0x00007ffee7020000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffedf9d3000)

vdso is injected into the address space by the kernel, so it can be ignored; ld-linux is the dynamic linker, which you cannot avoid anyway; pthread is strongly platform specific—you can consider it not suitable to decouple from the system, so it is best to load it dynamically; glibc is similar—otherwise it may core when you run it somewhere else.

Note that libm and librt can actually be linked statically, and the glibc-static package also provides the corresponding .a files. However, those files are not compiled with -fPIC, so they cannot be used to link a shared library; they can only be used to link executables. But as long as you manually build the corresponding version of glibc yourself, and copy out temporary build outputs named like libm_pic.a, you can use them as the libraries to link against. Note that these files are only temporary outputs, so you can only obtain them in this way.

So why doesn’t glibc provide .a files compiled with -fPIC? Because the whole point of the shared-library mechanism is to reduce library size and allow common functionality to be shared across system components. It is obviously discouraged to forcibly embed function binaries from the system’s default libraries into your own output, which completely goes against the design goals of shared libraries. But in many cases, I do need an so that is as general as possible so it can be used anywhere; that is where the techniques described in this article come in.

Example

Finally, here is a complete example of linker arguments. Suppose you want to link everything from libxxx.a, plus xxx.o, into a .so, and you want it to be very portable, i.e., it should only depend on an older glibc. Then you can link with the following arguments:

g++ -L. -shared -nodefaultlibs -o libxxx.so \
-specs=linker.specs \
-Wl,--wrap=memcpy \
-Wl,--wrap=__stack_chk_fail \
-Wl,--whole-archive \
libxxx.a xxx.o \
-Wl,--no-whole-archive \
-Wl,--start-group \
xxx.a .... \
$(gcc --print-file-name libgcc.a) \
$(gcc --print-file-name libgcc_eh.a) \
$(gcc --print-file-name libstdc++.a) \
libm.a librt.a libgomp.a xxx.a ... \
-Wl,--end-group \
-Wl,--as-needed -lc

All the above .a files must be built with -fPIC, which means you need to manually copy libm.a and librt.a from glibc build outputs, and you need to obtain libgomp.a from gcc build outputs. Also, the reason we do not use -static-libgcc and -static-libstdc++ but instead specify them manually like this is that librt.a will have symbol conflicts with those two. So it is better to put them all inside -Wl,--start-group -Wl,--end-group and adjust the order, making sure librt.a comes later.

libgomp.a with -fPIC

First, build gcc fully, with the following configure options:

../configure --enable-checking=release \
--enable-languages=c,c++  \
--disable-multilib \
--enable-shared \
--enable-host-shared \
--disable-tls

Note that you must manually disable TLS to prevent libgomp from using Thread Local Storage in the code. Otherwise, although static linking libgomp may succeed, it will fail at runtime and cannot be dlopened. This involves some implementation details of glibc when loading shared libraries that use the TLS mechanism.

Then go to <build-dir>/x86_64-unknown-linux-gnu/libgomp/. You will see many .o files. Note that the object files here are built without -fPIC. Open any .lo descriptor file and you can see something like:

# Name of the PIC object.
pic_object='.libs/error.o'
# Name of the non-PIC object
non_pic_object='error.o'

So the .o files compiled with -fPIC are actually located under <build-dir>/x86_64-unknown-linux-gnu/libgomp/.libs/. You only need to archive them into a new .a, which is the -fPIC-enabled libgomp.a we want.

Search Order

For the search order of shared libraries at link time, mainly refer to man ld: https://linux.die.net/man/1/ld.

For the search order at runtime, refer to man ld.so: https://linux.die.net/man/8/ld.so.

You can use readelf -d <exe> to inspect the .dynamic segment of an executable. NEEDED indicates which sos are required at runtime. If the so name does not contain an absolute path, the dynamic linker will search for it in a certain order. This search order is stored in the RPATH entry, where the $ORIGIN variable indicates the path location of the current binary at load time. You can set this order via the linker option -rpath.

For already-built binaries, you can use tools like patchelf to modify the dynamic linker and RPATH—this is not a very simple task, considering that the new path may be longer and requires additional handling of the ELF file. Common usage is as follows:

patchelf --set-interpreter /lib64/ld-2.17.so  ~/.dropbox-dist/dropbox-lnx.x86_64-59.4.93/dropbox
patchelf --set-rpath /lib64/ ~/.dropbox-dist/dropbox-lnx.x86_64-59.4.93/dropbox

Pinning Symbol Versions

GLIBC does not support linking to a specific version via link options. You can only achieve this using symver together with the linker’s wrap option. First, add the following code to your project:

:::C
#include <string.h>

void *__memcpy_glibc_2_2_5(void *, const void *, size_t);

asm(".symver __memcpy_glibc_2_2_5, memcpy@GLIBC_2.2.5");

void *__wrap_memcpy(void *dest, const void *src, size_t n) {
    return __memcpy_glibc_2_2_5(dest, src, n);
}

Then add to the linker (not compiler) options: -Wl,--wrap=memcpy. This makes all unresolved memcpy symbols resolve to the wrapper function __wrap_memcpy.

You can use objdump -p <so-name> or strings <so-name> | grep GLIBC to check all dynamic library versions a binary depends on. Use objdump -T <so-name> | grep GLIBC_2.14 to check which specific symbol in the binary references that GLIBC version.

GCC specs

The gcc program is essentially a compiler driver. Based on certain rules, it automatically invokes preprocessing, compilation, assembly, linking, and so on. These rules are stored in a file called specs. With each compiler release, a corresponding default specs file is shipped. Options in specs take precedence over all subsequently supplied command-line options. Therefore, if the compiler’s built-in rules contain entries that do not match expectations, we need to override them. gcc supports -specs to specify rule files on the command line, and you can specify multiple files; later ones can override earlier loaded ones.

Auto-generated __start & __stop for sections

If you use __attribute__ in code to place a function into a specific ELF section, and the section name you choose is a valid C identifier, then the linker will automatically generate the corresponding __start and __stop symbols for you, like this:

#include <stdio.h>
int i;
__attribute__((used, noinline, section("mysection")))
void test_func (void) {
    i++;
}
int main() {
    extern unsigned char __start_mysection[];
    extern unsigned char __stop_mysection[];
    printf ("Func len: %lu\n", __stop_mysection - __start_mysection);
    test_func ();
    return 0;
}

Note that you need used and noinline to ensure the compiler does not optimize your code away, and you must reference __start and __end somewhere so that the linker will auto-generate them when it notices missing symbols. In addition, this trick is not cross-platform. If you compile this code on macOS, you will get an error like:

test.c:3:40: error: argument to 'section' attribute is not valid for this target: mach-o section
      specifier requires a segment and section separated by a comma

Symbol Strip

Useful Parameters

GCC option summary: https://gcc.gnu.org/onlinedocs/gcc/Option-Summary.html.

Compiler

• static: On systems that support dynamic linking, this overrides -pie and prevents linking with the shared libraries. On other systems, this option has no effect.
• -nostdinc/--nostdinc++: Instructs the compiler to not search for header files in standard system directories. Tells the compiler to search for header files only in directories specified with the –I option, as well as in the directory of a current file, if appropriate.
• -nostdlib: Do not use the standard system startup files or libraries when linking.
• -nodefaultlibs: Do not use the standard system libraries when linking.
• -fno-builtin: Do not recognize built-in functions that do not begin with __builtin_ as prefix.
• -fno-asm: Do not recognize asm, inline or typeof as a keyword, so that code can use these words as identifiers.
• -rdynamic: This instructs the linker to add all symbols, not only used ones, to the dynamic symbol table. This option is needed for some uses of dlopen or to allow obtaining backtraces from within a program.
• -static-libgcc/-static-libstdc++: Attempt to link libgcc and libstdc++ statically.
• C_INCLUDE_PATH or CPLUS_INCLUDE_PATH or CPATH will set the include path for both C and C++.
• -fno-stack-protector: Disable the stack protector.

Linker

• --whole-archive/--no-whole-archive
  • For each archive mentioned on the command line after the --whole-archive option, include every object file in the archive in the link, rather than searching the archive for the required object files. This is normally used to turn an archive file into a shared library, forcing every object to be included in the resulting shared library.
• --start-group/--end-group
  • The archives should be a list of archive files. They may be either explicit file names, or -l options. The specified archives are searched repeatedly until no new undefined references are created.
  • Normally, an archive is searched only once in the order that it is specified on the command line. If a symbol in that archive is needed to resolve an undefined symbol referred to by an object in an archive that appears later on the command line, the linker would not be able to resolve that reference.
  • By grouping the archives, they all be searched repeatedly until all possible references are resolved. Using this option has a significant performance cost. It is best to use it only when there are unavoidable circular references between two or more archives.
• -Bstatic/-Bdynamic: Used to switch between static linking and dynamic linking on the command line; the library names in between are the ones to be linked statically.
• --wrap=xxx
  • Select a wrapper function for a specific symbol, typically used to select a particular memcpy version in GLIBC. Requires a corresponding implementation.
• --strip-all: Omit all symbol information from the output file.
• --strip-debug: Omit debugger symbol information (but not all symbols) from the output file.

Reference

• http://wiki.osdev.org/C%2B%2B_Exception_Support
• https://www.airs.com/blog/archives/56