python zmq many client to many server discovery message patterns

Question

Strugging on this problem for a while so finally asking for some help from the experts.

Language: python

The problem/setup:

I have many clients, client[n], client[n] .. etc
I have many servers, server[n], server[n] .. etc

Each server can plugin to 5 external ws connections. At any time I may need to open [x] ws connections; maybe 2, maybe 32, the total ws connections i need, thus servers needed, is dynamic...

Each client maybe connecting 1 ws connection from server[1], 1 ws connection from server[2] .. .etc

How I imagine the flow working

• New client[1] is loaded, needing 2 ws feeds
• New client[1] broadcasts [xpub/xsub ?] message to all servers saying, 'hey, I need these 2 ws connections, who has them?'
• Server[1] with the ws connections reply to client[1] (and only that client) - 'I got what youre looking for, talk to me'
• client[1] engages in req/reply communication with server[1] so that client[1] can utilize server[1]'s ws connection to make queries against it, eg, 'hey, server[1] with access to ws[1], can you request [x]' .. server[1] replies to client[1] 'heres the reply from the ws request you made'

tldr

• clients will be having multiple req/rep with many servers
• servers will be dealing with many clients
• client need to broadcast/find appropriate clients to be messaging with

Answer 1

The Zyre protocol is specifically designed for brokerless "gossip" discovery. Pyre (https://github.com/zeromq/pyre) is the Python implementation. It provides mechanisms for nodes to join a "discovery group" and share information. Among other things, it allows group members to WHISPER to individual members or SHOUT (multicast) to all members.

Zyre uses UDP broadcast beacons to initiate contact, so it is generally limited to a single subnet (UDP broadcast is generally not forwarded beyond a subnet). However, you could bridge a group across different subnets via your server(s) in each subnet (see below).

You could use zyre to distribute topology information (in this case, your server list) to your clients.

I have only played around with pyre a little, so I may not have all the details exactly right, but I would try to set it up like this:

1. Define a Zyre group.
2. Each server...
   • Joins the group.
   • Sets its server address (ip or fqdn, and maybe port) in its beacon header.
3. Each client...
   • Joins the group.
   • Reads server address from the HELLO messages it receives from servers.
   • Makes REQ connections to server(s).
   • Adds/removes server connections based on HELLO/LEAVE/AVOID messages received over time.

If servers are not in the same subnet (e.g., maybe they are in different AWS availability zones), you could preconfigure the servers to know what all the server IPs are, periodically verify that they are up (via REQ/REP or PUB/SUB between the servers), and SHOUT active-servers information to the local group. Clients could use this information to inform/adjust their list of active servers/connections.

I've thought about doing exactly the above, but it unfortunately hasn't risen above other priorities in the backlog, so I haven't gotten past the above level of detail.

Answer 2

I'll focus on the discovery problem. How do clients know which servers are available and which ws connections each one has?

One approach is to add a third type of node, call it broker. There is a single broker, and all clients and servers know how to reach it. (Eg, all clients and servers are configured with the broker's IP or hostname.)

When a server starts it registers itself with the broker: "I have ws feeds x,y,z and accept requests on 1.2.3.5:1234". The broker tracks this state, maybe in a hash table.

When a client needs ws feed y, it first contacts the broker: "Which server has ws feed y?" If the broker knows who has feed y, it gives the client the server's IP and port. The client can then contact the server directly. (If multiple servers can access feed y, the broker could return a list of servers instead of a single one.)

If servers run for a "long" time, clients can cache the "server X has feed y" information and only talk to the broker when they need to access a new feed.

With this design, clients use the broker to find servers of interest. Servers don't have to know anything about clients at all. And the "real" traffic (clients accessing feeds via servers) is still done directly between clients and servers - no broker involved.

HTH. And for the record I am definitely not an expert.