OCaml produce shared object with OCaml runtime linked in

Question

I'm trying to build a shared object out of an OCaml library that I can run under Python using the cdll interface in ctypes.

Is there a way to direct something in the OCaml toolchain to produce a shared object with its dependencies linked in?

I'm starting to suspect that OCaml simply does not support what I'm trying to do. I want to call an OCaml function, but then have it produce some kind of result and return control back to where it was called from.

Here's my OCaml library

(* foo.ml *)
let add x y = x + y

And the command I used to compile it on OS X. (using .so instead of .dylib is slightly violating the naming conventions of the platform, but ocamlopt will only accept an extension of .o or .so.)

$ ocamlopt -output-obj -fPIC -o foo.so foo.ml

succeeds and produces foo.so.

Here's my first attempt to load the shared object into a python program (python 3.6).

# foo.py
import ctypes

lib = ctypes.cdll.LoadLibrary('./foo.so')

print("Loaded")

It produces the following error when run

excerpt:

self._handle = _dlopen(self._name, mode)
OSError: dlopen(./foo.so, 6): Symbol not found: _caml_code_area_end

Full error:

Traceback (most recent call last):
  File "foo.py", line 3, in <module>
    lib = ctypes.cdll.LoadLibrary('./foo.so')
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ctypes/__init__.py", line 426, in LoadLibrary
    return self._dlltype(name)
  File "/usr/local/Cellar/python3/3.6.4_2/Frameworks/Python.framework/Versions/3.6/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen(./foo.so, 6): Symbol not found: _caml_code_area_end
  Referenced from: ./foo.so
  Expected in: flat namespace
 in ./foo.so

I looked around for the symbol in the compiler's directory ocamlopt -where.

$ find `ocamlopt -where` | xargs -I% bash -c 'nm % | sed -e "s|^|%:|"' | & grep _caml_code_area_end
/usr/local/lib/ocaml/libasmrun_pic.a: 0000000000000008 C _caml_code_area_end
/usr/local/lib/ocaml/libasmrun_pic.a:                  U _caml_code_area_end
/usr/local/lib/ocaml/libasmrunp.a: 0000000000000008 C _caml_code_area_end
/usr/local/lib/ocaml/libasmrunp.a:                  U _caml_code_area_end
/usr/local/lib/ocaml/libasmrun_shared.so: 000000000001ded0 S _caml_code_area_end
/usr/local/lib/ocaml/libasmrun.a: 0000000000000008 C _caml_code_area_end
/usr/local/lib/ocaml/libasmrun.a:                  U _caml_code_area_end
/usr/local/lib/ocaml/libasmrund.a: 0000000000000008 C _caml_code_area_end
/usr/local/lib/ocaml/libasmrund.a:                  U _caml_code_area_end

And tried loading /usr/local/lib/ocaml/libasmrun_shared.so first, giving

import ctypes

lib = ctypes.cdll.LoadLibrary('/usr/local/lib/ocaml/libasmrun_shared.so')
lib = ctypes.cdll.LoadLibrary('./foo.so')

print("Loaded")

which produced the error:

OSError: dlopen(/usr/local/lib/ocaml/libasmrun_shared.so, 6): Symbol not found: _caml_apply2
  Referenced from: /usr/local/lib/ocaml/libasmrun_shared.so
  Expected in: flat namespace
 in /usr/local/lib/ocaml/libasmrun_shared.so

The _caml_apply2 symbol is deep in the runtime of OCaml, and I wasn't able to find a file for it with the .so extension ... and cdll doesn't seem to expose the ability to load .as.

Étienne Millon Étienne Millon · Accepted Answer · 2018-02-15T08:21:18

For some reason, the -fPIC flag is not enough, you also need a PIC runtime. Previously, this required a special switch (such as 4.04.1+fPIC), but now you can select which runtime to link against.

The following works for me (Debian buster, opam 4.06.1+rc2 switch):

% ocamlopt -output-obj -runtime-variant _pic -o foo.so foo.ml
% python foo.py
Loaded

I'm trying to build a shared object out of an OCaml library that I can run under Python using the cdll interface in ctypes.

Note that you won't be able to just call the OCaml symbols from Python (or another interface), you'll need to write some C that sets up the runtime and manages the OCaml objects. A simple way to do that is to use Cstubs_inverted for this part.

OCaml produce shared object with OCaml runtime linked in

1 Answers