Overriding installation path in shared libraries built with automake/libtool during build for testing when DYLD_LIBRARY_PATH cannot be used

Question

Synopsis:

Starting with OS X 10.11, when System Integrity Protection is on, dynamic linker environment variables are not passed to subprocesses. In a project which produces two libraries, one dependent upon the other, test code (a shell wrapper around an executable) linked against the libraries fails to run, as the installation path is hardwired into the libraries, the libraries have yet to be installed, and the dynamic linker will try to find them using the hardwired paths in the libraries. Because DYLD_LIBRARY_PATH is not passed along to subprocesses (e.g., the executable run by the test wrapper script) it is not possible to direct the linker to look elsewhere.

Details:

The project consists of two libraries, one a shared library, the other a dlopen'able module which loads the shared library.

Here's the automake snippet:

#-------------------
# shared library
lib_LTLIBRARIES           += %D%/liblua_udunits2.la

%C%_liblua_udunits2_la_CFLAGS = $(UDUNITS2_CFLAGS) $(LUA_INCLUDE)
%C%_liblua_udunits2_la_SOURCES = %D%/lua_udunits2.c

#-------------------
# dlopen'able module

luaexec_LTLIBRARIES = %D%/udunits2.la

%C%_udunits2_la_CFLAGS  = $(UDUNITS2_CFLAGS) $(LUA_INCLUDE)
%C%_udunits2_la_LIBADD  = $(UDUNITS2_LIBS) %D%/liblua_udunits2.la
%C%_udunits2_la_SOURCES = %D%/udunits2.c
%C%_udunits2_la_LDFLAGS = -L%D% -module

This produces

• liblua_udunits2.dylib
• udunits2.so

When the linker builds udunits2.so, it records the install name found in liblua_udunits2.dylib,

% otool -D lua_udunits2/.libs/liblua_udunits2.dylib

lua_udunits2/.libs/liblua_udunits2.dylib:
/usr/local/lib/liblua_udunits2.0.dylib

into udunits2.so.

% otool -L lua_udunits2/.libs/udunits2.0.so

lua_udunits2/.libs/udunits2.0.so:
/usr/local/lib/libudunits2.0.dylib (compatibility version 2.0.0, current version 2.0.0)
/usr/lib/libexpat.1.dylib (compatibility version 7.0.0, current version 8.0.0)
/usr/local/lib/liblua_udunits2.0.dylib (compatibility version 1.0.0, current version 1.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)

The test code is a shell script which calls an executable which loads udunits2.so, which then attempts to load liblua_udunits2.dylib. This fails, as it is using the install name for liblua_udunits2.dylib, and it is not yet installed.

I've worked around this for previous versions of OSX by adding

TESTS_ENVIRONMENT += DYLD_LIBRARY_PATH="$(top_builddir)/$(PACKAGE_NAME)/.libs"

for the test code.

However, since OS X 10.11 (El Capitan), if System Integrity Protection is on, dynamic linker environment variables are not passed to subprocesses in certain cases. In my case the test code is a script which calls an executable, so the DYLD_LIBRARY_PATH is not passed to the executable, and I cannot override the library load path. (The script is part of a third party testing framework which is external to my code, so modifying it to explicitly set DYLD_LIBRARY_PATH isn't an option).

Question:

Is there a way to have automake/libtool use the build directory path to create the libraries so that testing can proceed, and then relink them using the installation path during installation?

Answer 1

The least intrusive means of doing this seems to be to use install_name_tool (as mentioned by I'L'I in the comments in the question) with hopefully only a minor reliance upon how automake implements things.

My solution is to create a target which depends upon the dlopen'able module and changes the path of the dynamic library dependency from the installation directory to the build directory. This allows tests which link against the module to successfully load the dynamic library. Upon installation, the dependency path is restored to the installation path.

The modified Makefile.am looks like this:

#-------------------
# shared library
lib_LTLIBRARIES           += %D%/liblua_udunits2.la

%C%_liblua_udunits2_la_VERSION_MAJOR = 0
%C%_liblua_udunits2_la_VERSION_MINOR = 0

%C%_liblua_udunits2_la_CFLAGS = $(UDUNITS2_CFLAGS) $(LUA_INCLUDE)
%C%_liblua_udunits2_la_SOURCES = %D%/lua_udunits2.c
%C%_liblua_udunits2_la_LDFLAGS = -version-info $(%C%_liblua_udunits2_la_VERSION_MAJOR):$(%C%_liblua_udunits2_la_VERSION_MINOR)


#-------------------
# dlopen'able module

luaexec_LTLIBRARIES = %D%/udunits2.la

%C%_udunits2_la_VERSION_MAJOR = 0
%C%_udunits2_la_VERSION_MINOR = 0

%C%_udunits2_la_CFLAGS  = $(UDUNITS2_CFLAGS) $(LUA_INCLUDE)
%C%_udunits2_la_LIBADD  = $(UDUNITS2_LIBS) %D%/liblua_udunits2.la
%C%_udunits2_la_SOURCES = %D%/udunits2.c
%C%_udunits2_la_LDFLAGS = -L%D% -module -version-info $(%C%_udunits2_la_VERSION_MAJOR):$(%C%_udunits2_la_VERSION_MINOR)

if HOST_OS_IS_DARWIN

noinst_DATA = %D%/fix_install_name

%C%_DYLIB_NAME  = liblua_udunits2.$(%C%_liblua_udunits2_la_VERSION_MAJOR).dylib
%C%_MODULE_NAME = udunits2.$(%C%_udunits2_la_VERSION_MAJOR).so

%C%_LIBSDIR = $(abs_builddir)/%D%/.libs/

%D%/fix_install_name : %D%/udunits2.la
    install_name_tool               \
        -change                 \
        $(libdir)/$(%C%_DYLIB_NAME)     \
        $(%C%_LIBSDIR)/$(%C%_DYLIB_NAME)    \
        $(%C%_LIBSDIR)/$(%C%_MODULE_NAME)
    touch $@

install-data-hook :
    install_name_tool                   \
        -change                     \
        $(%C%_LIBSDIR)/$(%C%_DYLIB_NAME)            \
        $(libdir)/$(%C%_DYLIB_NAME)         \
        $(DESTDIR)$(luaexecdir)/$(%C%_MODULE_NAME)

endif

Things of note:

• The library version is now explicitly specified so that the names of the dynamic library and the dlopen'able module are known deterministically
• The code uses the fact that the libraries are staged in .libs
• The code explicitly uses the .so and .dylib extensions. I don't believe there's a way of getting this info out of libtool.