Explanation sought: libtool, automake, shared libraries (and Fortran)

Question

The problem I had is solved. I'm posting this to solicit an explanation as to why the solution actually works. I've gotten great feedback here before.

I have a legacy code base that used a very simplistic build system, and my project is to migrate that to Autotools for customization and, particularly, building shared libraries. The main library is written in C, but must also be linkable from Fortran (for legacy purposes), and is distributed with some test codes in F77. The authors organized the source code into modules...

src_module1/
src_module2/
...
testc/
testf77/

Their built the library lib/libmain.a by compiling code in the src_*/ directories and archiving the objects with ranlib.

My first approach was to build a shared library from each src_*/ separately and "link" all of these into one shared library. Using Autotools, the src_module1/Makefile.am would contain

noinst_LTLIBRARIES = libmodule1.la
libmodule1_la_SOURCES = ...

and so on for the other modules, and finally the lib/Makefile.am would need only:

lib_LTLIBRARIES = libmain.la
libmain_la_SOURCES =
libmain_la_LIBADD = $(top_srcdir)/src_module1/libmodule1.la ...

This seemed to work perfectly. However, when the code in testc/ was compiled and linked against libmain.la, a "symbols not found" error was issued.

Thinking that this was an issue with libtool or shared libraries, I tried building static only, basically changing all .la to .a and all _LTLIBRARIES to _LIBRARIES. Same problem. This time, however, noticing the error "ranlib: warning for library: libmain.a the table of contents is empty (no object file members in the library define global symbols)" when trying to link libmain.a itself.

The solution that I found seems like a hack. I did not build Makefiles for any of the src_*/ directories, but instead used only for the lib/ directory and its Makefile.am had the lines:

lib_LTLIBRARIES = libmain.la
libmain_la_SOURCES = [all sources from all ../src_modules/ ]

This worked. The compiled programs in testc/ linked against libmain.la without issue. One of the "modules" is a set of Fortran bindings that wrap other C functions in the library. Even the Fortran codes in testf77/ linked against libmain.la properly.

Could someone carefully explain what happens when libtool builds a shared library? Or even when building a static library? Why is it that several static libraries can't be linked together to make one static library? Why are symbols only available when libtool/ranlib builds the library "from sources"? And what about installing a shared/static library, i.e. moving it to the /usr/local/lib --- what happens there? The Wikipedia article on static and shared libraries isn't really detailed enough for me.

I do appreciate all efforts to make sense of my longwinded question.

Answer 1

What you first tried ought to work. I am using this kind of setup all the time (in a C++ context). It's also documented, and part of the Automake test suite (although maybe not with Fortran).

A libtool library that is not installable, i.e., one declared with noinst_LTLIBRARIES, is called a libtool convenience library. That noinst_ makes a big difference in what is built. Even if Libtool is configured to build shared libraries, a libtool convenience library is not actually a shared library: it is just a set of object files (compiled as PIC so that they can be latter be used in a shared library) stored in an archive. You can use a libtool convenience library anywhere using this set of objects would make sense, e.g., to build a shared library.

When multiple libtool convenience libraries are LIBADDed to an installable libtool library (such as your libmain.la), Libtool has to unpack the archives containing the objects of each convenience library and link them into the final library.

There is a trap that is worth noting here: when building a shared library out of convenience libraries, if the _SOURCES variable is empty Automake does not know which linker to use and default to the C linker. If you want to trick Automake into using the linking rule for some specific language, you can declare a nodist_EXTRA_..._SOURCES source file that do not have to exist. (See the Libtool Convenience Libraries section of the Automake manual for an example.)

Maybe that was your problem? If you have some Fortran files in the sources of some of your modules (your description suggests these are only C files), the Fortran linker will be used to build libmain.la only if a Fortran file appears in the source files declared for that libtool library. And the C linker will be used when libmain_la_SOURCES is empty.

Otherwise, I have no idea why it didn't work.

There is an small error in your example:

libmain_la_LIBADD = $(top_srcdir)/src_module1/libmodule1.la

should be

libmain_la_LIBADD = $(top_builddir)/src_module1/libmodule1.la

because the library is not created in the source directory. However I assume this is just a typo, and you won't see the difference unless you do a VPATH build or run make distcheck.

---

Your second try, using _LIBRARIES without Libtool is not expected to work.
_LIBRARIES can only be used to declare static archives, and in this case _LIBADD may only contain object files, not other static archives. Unpacking an archive to reuse its objects into another archive can be tricky to do portably. Automake's answer to this problem has always been: install Libtool and use _LTLIBRARIES (Libtool can be configured to build only static libraries).