rabbitmq event driven microservices

Question

Microservice architecture and sharing common application data.

Scenario being: There are today 17 microservices for some online social media service and 9 of them need to know who is connected to who in order for their function to work. To prevent each service constantly asking the "authentication" or "connections" microserice for the list, all services register to recieve a copy of the connections per user and store in a cache.

A proposal for the mechanism to deliver the data, or instruction to fetch data could be rabbitmq.

However, each microservice is a cluster of docker containers orchestrated by k8s for scalability.

Each container registers to listen to a collection of exchanges they are interested in... so for the "news feed" service that could be say 5 connections...

Below is an illustration of the proposed setup:

• T1 - user A accept a friend request
• T2 - The connections service (MS1) makes the connection in its primary database
• T3 - MS1 published to a rabbitmq exchange the said event
• T4 - rabbitmq exchange emits to all Q's (ie all other microservices registered)
• T5 - All the nodes within the MS2 cluster pickup the event and act... their action (in this case) will be to update the cache of the friend connections.
• T6 - user A requests the data for their newsfeed, MS2 now queries its database with the use of its local cache

This is all good:

• The connection service didn't know or care who got the data, only that it should emit to 1 exchange via the single rabbitmq entry point
• The developer of MS2 only needed to know about the location of the rabbitmq instance
• The developer of all the other services the same.. they handle the data in their own brilliant way.

The 1 exception is.. there were 3 instances of MS2 so that would be 3 database writes.. if the system scales to 10 that would be 10 db writes etc etc.

Question How is this problem bypassed... how to ensure only 1 of the MS2 instances will act?

Should the newsfeed microservice be delivered with its own internal q system to manage the data from the exchange? Is it possible to route all the messages via the load balancer so only 1 instance of MS2 gets a message? I don't want to start managing lots of lots of queues by hand as this will be a pain and defeat the simplicity of the exchange design.

Answer 1

So, all instances if M2 will share a queue and work using the competing consumer pattern, every messages is consumed once and if all instance of M2 goes down the queue grows until they come back up again.

M2, M3 and M4 will each create ONE queue for what M1 publishes. Let's name them them M2_from_M1, M3_from_M1 and M4_from_M1. They will all also create a binding against the exchange M1 uses and on the routing key for this message.

Now, instances of M2 will all consume from M2_from_M1, instances of M3 will all consume from M3_from_M1 and so on.

If all instances of any of these are down it's queue will start to fill up but that is fine since it will be consumed later.

Regarding the overall architecture. Try first with actually making the call between M2 and M1, access time between pods is probably very fast and you could probably cache both in M1 and in M2 for a while. Worst outcome is that you see news from people you no longer follow, or that you don't get news from new contacts.