Is it possible to PULL a ZeroMQ PUSH Socket but with a filter just like PUB/SUB?

Question

I'm using a PUSH socket to send messages to workers, which are receiving through PULL, so the messages get uniformly distributed amongst workers. However, I'd like to use some workers to process specific messages. In the PUSH/PULL method, there's no special treatment for each worker, they're all equal. I though of using PUB/SUB then, because of the filter, but there's no way to automatically distribute to workers. If a set of workers is meant to process message A, they'll all receive it. That's not good.

The perfect method would be to a set A workers be able to PULL messages only meant to them, and the set B PULL messages meant to B. I could, for example, set a port for the set A and another for the set B, but it'd be so much elegant to have this solution for one single socket, and I also see no reason why it couldn't be done by the developers.

Is there a way to do it with the existing version of ZeroMQ?

The basic idea is: just one PULL, which would send enveloped messages (just like in PUB), and workers could PULL messages with this envelope. See that the workers could PULL every message and see if it's meant for them, but this would require extra processing and rebroadcast of the received message so it's not discarded.

Answer 1

You could do this:

PUB the original message to two SUB (A/B) that filter the messages appropriately. And then those two SUB sockets can be attached to two PUSH sockets that send to the appropriate PULL workers.

I use this pattern in production and it works well.

Answer 2

Essentially I think you are going to have to implement this functionality yourself. I imagine it looking like this...

Implement the PUSH/PULL architecture like normal. However, if a worker gets a message it cannot process, it sends back a message to the source telling it to give that message to the next worker. This would essentially keep happening until the message gets to a worker that can process the message. To make this work, you will need to have a way of sending a message back to the source to let it know whether you can or cannot process the message. To do that, you can look into the ROUTER/DEALER architecture, or simply setup the source as a SUB which listens for messages from the workers indicating whether they could/could not process the message.