ZeroMQ + NodeJs: Messages not received by subscriber

Question

My publisher (zpub.js) publishes in a loop as shown below.

async function publishLoop() {
    let payload = []
    _.forEach(array, (a) => {
       // process a here to generate someKey and someValue
       payload.push({someKey:someValue})
    })

  return Promise.all(payload.map(async (p) => {
    await zmqp.publish({t:'topicString', m:p})
  }))
}

zmqp.publish is simply the following.

async publish(payload) {
    // this.sock is just a bind to tcp://127.0.0.1:4030
    await this.sock.send([payload.t, JSON.stringify(payload.m, null, null)])
    return Promise.resolve()
}

My subscriber (zsub.js) is a version of code as seen on the ZeroMQ website.

const zmq = require("zeromq")
const mom = require('moment-timezone')

async function run() {
  const sock = new zmq.Subscriber

  sock.connect("tcp://127.0.0.1:4030")
  sock.subscribe("topicString")

  for await (const [topic, msg] of sock) {
    console.log(`${mom().tz('Asia/Kolkata').format('YYYY-MM-DDTHH:mm:ss.SSS')}`)
  }
}

run()

I start my subscriber as node zsub.js > out.
I launch my publisher as node zpub.js. All messages received successfully.
The zpub.js process ends but zsub.js keeps running. When I re-run node zpub.js, not a single message is received by the subscriber. The number of records in out remains unchanged.
Running zpub.js once or twice again seems to deliver messages (the recent ones; not the earlier ones as seen by timestamp) to subscriber.

Thus, I am not sure, what is to be done on the pub/sub side so that messages aren't 'lost'. Please advise.

"running zpub.js once or twice again seems to deliver messages". I don't think communication is restored because of restarting a few times but because enough time has passed. What happens when you wait "a long enough time" between stopping and restarting your publisher once? — rveerd
I ran the tests again. Average of 90 publications per node zpub.js with 20 seconds gap and messages are getting dropped. — cogitoergosum

rveerd rveerd · Accepted Answer · 2020-08-26T07:19:47

Pub-Sub is inherently unreliable because there is no feedback from the subscribers to the publisher to acknowledge that messages have been received. The guide describes this in detail and provides some solutions. One of the solutions is not to use Pub-Sub but, for example Router-Dealer. It depends on your use-case whether this is a viable alternative.

Regarding your specific issue, the subscriber eventually determines that the connection to the publisher is lost and will try to reconnect until a connection is re-established. Depending on the timing, the subscriber can miss initial (or all) messages send by the publisher.

In general, Pub-Sub works best if the publisher is the stable part of the communication (stays up and online, like a server) and the subscribers can come and go (like a client).

ZeroMQ + NodeJs: Messages not received by subscriber

1 Answers