"Best Practice" for a clojure library that uses native libs?

22
votes

While this may appear subjective, there is a concrete example that I'd like help resolving. This is related to an issue with the Overtone Clojure library https://github.com/overtone/overtone/issues/274 which seems like there should be a "Best Practice" for Leiningen and apply to more libraries than just Overtone.

Overtone is a clojure library that is meant to be used from within other projects. Overtone requires native libraries to work, so it uses :native-path "native" in the project.clj https://github.com/overtone/overtone/blob/master/project.clj#L69 in order to get a proper path for the native scsynth libraries [overtone/scsynth "3.5.7.0"] that are used.

However, I believe that this resets the incoming path from a project that depends on the Overtone library. See the issue for some background, but basically after depending on [overtone "0.9.1"] in a project.clj (System/getProperty "java.library.path") returns only the current native path and the project using Overtone cannot pass in a path to any local libraries.

So, the question is--how can a dependent project mix local native libraries with Overtone? Should Overtone or the dependent-project adjust its project.clj settings? How?

3
clojureleiningen
Did you try setting :native-path in your own project.clj? It looks like lein will join this with the dependency's native-path. – Alex
Interesting...initial testing seems positive, but java.library.path only reports my new path, not Overtone's path. Will circle back after more checkout, thanks! – Roger Allen
Nope...not sure why it may have worked initially, but with :native-path "leaplib" and a library in "leaplib/macosx/x86_64" it fails to find overtone scsynth libs. Setting :native-path appears to override, not join. – Roger Allen
I've updated github.com/overtone/overtone/issues/274 with some recent findings, but I still consider this an unsolved problem... – Roger Allen
People who are upvoting, consider adding your voice to this Leiningen issue github.com/technomancy/leiningen/issues/1385 – Roger Allen

3 Answers

3
votes

I don't know about "best", but here is a practice that has worked successfully for me in at least four projects now, three of which were "real", commercial ones. Although I've open-sourced only a specific case for ZeroMQ, I believe the principles are generic and should work for any native libraries. Most of the code could easily be reused, too, and it is licensed under Eclipse, so feel free to take it if you want. I haven't had a need for a more generic version of a native-including library, but I believe it could easily be extracted.

The problem I had with the standard solutions (lein's :native-path, JVM args, etc.) was that I wanted a portable solution for uberjar distribution that does not require the user to install anything else, so instructions such as "download the uberjar, install libzmq-dev from your package manager, and then launch the uberjar" were out of the question.

The principle is pretty simple: I bundle the native libraries within the library jar, for all supported platforms. That way:

  • I, as the library author, control specifically which versions are used. No surprises there.
  • Including the library from another project can be done without any awareness that the library relies on native libs.
  • Similarly, uberjar distribution, whether from leiningen or maven, work seamlessly.

I'm not relying on any of the OS linking strategies either, for I found it hard to make them work reliably. So what I did was that I included all of the required native libraries that are not guaranteed to be present on a running system within the jar, and then, at runtime, the library:

  • dynamically finds out what platform it is running on;
  • extracts the native libs from its own archive towards the system-specific temp directory if it is not already there (based on a SHA; this was necessary to avoid accumulation on platforms that do not properly clean their temp dirs -- that is, Windows);
  • "manually" load the native dependencies in the correct order, so dynamic linking does not try to resolve libraries on the host OS path.

There are of course downsides to that approach:

  • As the build process for a single artifact involves three different OSes (I support Linux, Windows and OSX) and 2 architectures (i386 & x86_64), the build process is hard to automate and requires access to the relevant machines;
  • Because specific versions of the native libs are bundled, library users cannot update them; in security-sensitive environments, this may be unacceptable;
  • As a user of this library, you have to trust me that the bundled native libs are indeed what I claim they are, and have indeed been built through the process I describe in the source code, without any more tampering;
  • As I know practically nothing about linking dynamic libraries, there may be other problems with this approach, though they haven't bitten me yet.
2
votes

I released clj-nativedep via Clojars which can help with this problem. The library provides the ability to quickly identify a normalized name for the current system architecture, and can load any selected resource (within the jar or classpath) into the runtime environment.

See: https://github.com/rritoch/clj-nativedep

This system was specifically made for my WarpCTL project which utilizes a great deal of native code generated with swig. Due to how Clojure classloading is handled the native dependencies need to be loaded via a static class constructor, you can see an example at https://github.com/rritoch/WarpCTL/blob/master/extra/JADL-SDK/build/java/src/com/vnetpublishing/swig/adl/jadl_sdk.java#L13. For that project I build the java code into a JAR and add clj-nativedep and the jar as a dependency. It should be possible to load resources this way from pure clojure applications but for the best performance it needs to be loaded from a static class constructor.

0
votes

I also had this problem once, so i made an unorthodox lein plugin that solves just that once and for all with a couple of lines added to your project.clj: https://bitbucket.org/noncom/nativot

WARNING: it is extremely unwelcomed by the Clojure way since it breaks all the concept of repeatability and stuff, allowing you to pack arbitrary jars, resources and other files into your resulting jar and just makes it work.