How to structure a plain zeroMQ one way Push-Pull pattern with N-producers (connect) and 1-consumer (bind) - MT4 to Python

Question

Multiple Users will push data from Metatrader 4 to a Python backend. Thus the amount of running instances of Metatrader 4 (= producers) is dynamic. Each instance of MT4 will run on the users local machines, create their own zmq.context and zmq.socket (push). So we end up with a N-to-1 structure basically:

|-------------------Producer1-----Producer2-----Producer3-----Producer4-----ProducerX-------------------------------------------------|-------------------|----------------|-----------------|-------------------|------------------------------
........................................|-----------------Consumer/Backend------------------|...........................................

That being said, the consumer will run 24/7 and waits for data from whatever producer..E.g. if there are 160 users online to use the service the consumer will have to process data input from those 160 nodes so on and so forth. The order of received messages doesn't matter and if some messages drop occasionally it won't matter as well.

Current status:
Right now my code (see below) runs very well using one Metatrader 4 instance on my own local machine. But what happens if there are X instances of MT4 with all of them having their own zmq contexts and push sockets within the local metatrader 4 instance to send data? Will it just work as intended as well?

Code producer end (Metatrader 4 instance) which every user will run on his local machine:

extern string ZEROMQ_PROTOCOL = "tcp";
extern string HOSTNAME = "localhost";
extern int PUSH_PORT = 32225;

// CREATE ZeroMQ Context
Context context(PROJECT_NAME);

// CREATE ZMQ_PUSH SOCKET
Socket pushSocket(context, ZMQ_PUSH);

//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+

int OnInit()
  {

//---

   EventSetTimer(1);     // Set Second Timer as push intervall

   context.setBlocky(false);

   // Send data to PULL_PORT that consumer is listening to.
   Print("[PUSH] Connecting MT4 Dashex Feeder to Dashboard on Port " + IntegerToString(PUSH_PORT) + "..");
   pushSocket.connect(StringFormat("%s://%s:%d", ZEROMQ_PROTOCOL, HOSTNAME, PUSH_PORT));

   pushSocket.setSendHighWaterMark(1);
   pushSocket.setLinger(0);   

   return(INIT_SUCCEEDED);
  }

//---

void OnDeinit(const int reason)
{

//---

   Print("[PUSH] Disconnecting MT4 Dashex Feeder on Port " + IntegerToString(PUSH_PORT) + "..");
   pushSocket.disconnect(StringFormat("%s://%s:%d", ZEROMQ_PROTOCOL, HOSTNAME, PUSH_PORT));

   // Shutdown ZeroMQ Context
   context.shutdown();
   context.destroy(0);

   EventKillTimer();
}

[...]

pushSocket.send(StringFormat("%s", account_info, true));

Python backend consumer:

# create zeroMQ Pull Socket to fetch data out of the wire
context = zmq.Context()
zmq_socket = context.socket(zmq.PULL)
zmq_socket.bind("tcp://*:32225")
time.sleep(1)

while True:
    # check 24/7 for available data in the pull socket
    try:
        [...]

Questions:

1.) The current code creates a new context and socket per each metatrader 4 instance running on a users local machine and closes them again once MT4 is closed. Can multiple MT4 instances/push sockets send data to the same consumer at once? (this is my desired outcome)

2.) Additionally can each terminal instance use the same socket ports or does each terminal instance need a unique one?

3.) Should I maybe use a different pattern?

In advance thank you very much for your thoughts and input.

James Harvey James Harvey · Accepted Answer · 2019-12-12T10:02:05

You only need to share a context if you are using inproc transport. As you are using tcp it will work as you expect.
The terminals will connect() with the port specified by the python backends bind(). So yes all the terminals will connect to port 32225 as specified in your code.
The push/pull pattern seems like a good choice and should work fine. If you are concerned about the overhead (many connections) on you python backend you could place a zeromq proxy (PULL/PUSH) proxy process in between the terminals and the backend.

How to structure a plain zeroMQ one way Push-Pull pattern with N-producers (connect) and 1-consumer (bind) - MT4 to Python

1 Answers