Connecting to the CPLEX remote API with julia

Question

I'm setting up a parallel optimisation environment using IBM CPLEX 12.9, Julia Language v1.1.0 and JuMP. To start a local optimisation I'm currently using the library CPLEX.jl that provides the connection (using C calls on background) to optimise some model locally. Let's call this machine A.

However, I'm trying to start an optimisation in a remote machine which means that when I start an optimisation on A, Julia will call the CPLEX installed on the machine B (which has more memory, cpus, etc).

Looking the CPLEX documentation I've seen that for a local optimisation we call the function

CPXopenCPLEX(int * status_p)

given by the lib libcplex1290.so. For a remote connection, CPLEX provides another interface to connect to external servers by the function

CPXopenCPLEXremote(char const * transport, int argc, char const *const * argv, int * status_p)

The package CPLEX.jl supports only local optimisation and it uses the CPXopenCPLEX() function. Looking for this package, the connection with the local CPLEX installation is made by the following command:

ccall(("CPXopenCPLEX",libcplex),Ptr{Cvoid}, (Ptr{Cint},),stats)

where libcplex="/opt/ibm/ILOG/CPLEX_Studio129/cplex/bin/x86-64_linux/libcplex1290.so", and stats is an Array{Int32,1}. This command is found at the file cpx_env.jl of the package CPLEX.jl.

What I've tried is to implement a similar function that will call CPXopenCPLEXremote insteat of CPXopenCPLEX with the correct values. My Julia1.1 code is the following:

const libcplex = "/opt/ibm/ILOG/CPLEX_Studio129/cplex/bin/x86-64_linux/libcplex1290remote.so"

 argv=["/usr/bin/ssh", "IP_OF_REMOTE_MACHINE","/opt/ibm/ILOG/CPLEX_Studio129/cplex/bin/x86-64_linux/cplex", "-worker=process"]


ret= ccall(("CPXopenCPLEXremote",libcplex),Ptr{Cvoid}, (Ptr{Cchar},Cint,Ptr{Ptr{Cchar}},Ptr{Cint},),"processtransport",Int32(4),argv,stats)

The problem is ret=Ptr{Nothing} @0x0000000000000000 with means that the connection did not succedd.

I'm quite sure that the problem is in the way that I'm giving the arguments to ccall() to call CPXopenCPLEXremote.

Could someone with experience in this tye of call help me with the parameters?

I'm also configuring an automatic identification for the ssh connection. For now I've to inform my user and password on each ssh connection from the machine A to the remote machine B. (I'll update this question later)

Thank you all for any help.If it works, I'm going to create the lib CPLEXremote.jl for the community.

best regards, Isaias

What is the value of status_p after the call to CPXXopenCPLEXremote? That may give you an error code that will help diagnose the issue. In case you haven't seen it, there is extensive documentation about remote object here. — rkersh
BTW, It sounds like your follow-up question about configuring an automatic ssh connection should be asked in a new question rather than appended here. — rkersh

Daniel Junglas Daniel Junglas · Accepted Answer · 2019-05-25T13:03:37

Many things could go wrong here. I don't know Julia but here are the things that could try outside of Julia to sort this out:

You definitely need passwordless ssh connection. There is no way you can supply username/password with the CPLEX remote object API. This is mentioned in the documentation here.
On both machines make sure that not only CPLEX is installed but also that the folder that contains the various libcplex*transport.so and libcplex*worker.so libraries is in LD_LIBRARY_PATH. The remote object code has to load these libraries dynamically at runtime.
For debugging purposes set environment variable ILOG_CPLEX_REMOTE_OBJECT_TRACE to 99. This should give more information about the error that happens.
Try adding either -stdio or -namedpipes=. to the command line.
Take a look at the example cplex/examples/src/remotec/parmipopt.c. This basically does what you plan to do. It also involves user functions, so it is a bit more complicated than what you plan.
Look at example cplex/examples/src/remotec/parbenders.c this does more complicated things in the solution process but the setup of the remote solvers is pretty simple. You can run this example by going to cplex/examples/x86-64_linux/static_pic and saying make -f Makefile.remote remote-run-parbenders. It is a good idea to start with that and trying to modify it so that it does not only run on your localhost but actually connects to the remote machine correctly. This takes Julia out of the picture. Once you have this working go back to Julia and figure out how to invoke CPLEX from there.

Connecting to the CPLEX remote API with julia

1 Answers