Integrating OCaml in python - How to hold an ocaml session from python?

Question

I would like a python GUI to have an OCaml process in the background. I would like to keep a single session throughout the program's lifetime, and depending on user inputs, call some OCaml commands and retrieve OCaml's output. Some OCaml variables and structures may be defined along the way so I would like to maintain a single ongoing session.

My solution was to hold an OCaml toplevel process using popen and interact with its stdin and stdout. This works purely for me for several reasons: 1. I don't know when is the OCaml calculation done and can't tell if it's output is complete or there is more to come (especially so if the evaluation takes some time, and if multiple OCaml commands were invoked). 2. I have no inherent way of telling whether the OCaml command ran smoothly or maybe there were OCaml warnings or errors. 3. I lose the structure of OCaml's output. For example, if the output spreads over several lines, I can't tell which lines were broken due to line size, and which were originally separate lines.

I know there are some discussions and some packages for combining python with OCaml, but they all run python commands from OCaml, and I need the opposite.

Answer 1

If you want to run OCaml in a separate process, you need to wrap your OCaml calls in a function that collects all results and returns them in a useful serialization format, e.g. JSON. Alternatively, you could try an write a Python C extension that use OCaml toplevel as a library, but that's likely a fair bit of work.

Answer 2

If you need a strong communication between your Python and OCaml code, there should indeed be two separate "master" processes (think of them as network nodes).

As @Sven Marnach already mentioned, a good option to implement this is to link these two processes via JSON-based protocol.

A more convenient approach would be to use Google's gRPC framework (https://grpc.io/) and communicate via Protobuf messages (https://developers.google.com/protocol-buffers/). The framework is very handly. Unfortunately, there is no support for OCaml yet, but I think you can thin wrap your OCaml main into thin Python layer, or translate it to JS. Then, all you need to do is just to connect your functions to gRPC interfaces.

Here is how the system would look like:

+----------+   +------+   +---Thin Python wrapper / JS wrapper---+
|   Your   |   |      |   |  +--------------------------------+  |
|  Python  |<->| gRPC |<->|  |        Your OCaml app          |  |
|   app    |   |      |   |  +--------------------------------+  |
+----------+   +------+   +--------------------------------------+

P.S. I'm using the same approach in a problem similar to yours (but GUI is in Java). I'd say that it is very convenient, fast to develop, and easily extendable.

P.P.S. You are not alone in this :). Here is an interesting excerption from a paper by a (former?) Google employee (https://arxiv.org/abs/1702.01715):

Software engineers at Google are strongly encouraged to program in one of five officially-approved programming languages at Google: C++, Java, Python, Go, or JavaScript​.

Interoperation between these different programming languages is done mainly using Protocol Buffers​. Protocol Buffers is a way of encoding structured data in an efficient yet extensible way. It includes a domain-specific language for specifying structured data, together with a compiler that takes in such descriptions and generates code in C++, Java, Python, for constructing, accessing, serializing, and deserializing these objects. Google’s version of Protocol Buffers is integrated with Google’s RPC libraries, enabling simple cross-language RPCs, with serialization and deserialization of requests and responses handled automatically by the RPC framework.

Answer 3

As a complementary remark, it is perfectly possible to run a separated toplevel process and send to it input phrases and read the corresponding output. The trick to detect the end of a toplevel output is to add a guard phrase after every input phrase: rather than sending just f ();; to the toplevel process, one can send f ();; "end_of_input";; and then watch for the toplevel output corresponding to "end_of_input";; (aka - : string = "end_of_input"). My experience is that errors and warnings are generally quite easy to detect or parse from the toplevel output; so the only missing point is the formatting of the code.