event driven microservices architecture

0
votes

I'm new to event-driven microservices, and rather new to microservices themselves. The system I'm working on, which isn't that large, processes a bunch of files then depending on the data different services are called. So it seems to me that a good idea would be, instead of having a service call other services to do the job, to send those messages to a pub/sub queue, process them and send them to the relevant topic, then each service would subscribe to one or several of those topics, and every few minutes each one of those services will pull the messages they are subscribed to and do their magic. This services could also be exposed through rest in case you want to force their execution without having to wait.
What's the best approach for a design like this? What am I missing? Is there something like this that could be implemented with spring? (I'm thinking each micro service could be a spring-boot app). I was planning on using GCP's Pub/Sub, but I heard of Apache Kafka, however I don't know how Kafka may help where Pub/Sub can't or how Pub/Sub can be integrated with it.

1
restapache-kafkamicroservicesgoogle-cloud-pubsubevent-driven
Idea of sending those messages to a pub/sub queue seems fine, better than rest I think. You could use Kafka. You would have different producers in the service which processes a bunch of files and in each other service you would have desired consumer. It would be asynchronous. Different services (all of them) could be implemented with spring-boot easily. - vanillaSugar
I'll be working under GCP... Are there advantages of Kafka over GCP's Pub/Sub? Is there a way to leverage Pub/Sub with Kafka? - CCC
I don't have any experience with GCP so I don't know, sorry. Can you use GCP's Pub/Sub for your services as I wrote in the previous comment? You would simply have producer which would write to different queues (topics) and in each other microservice you would have consumer which would consume desired queue (topic). This scenario is very common with Kafka. - vanillaSugar
At very high-level Apache Kafka and Google PubSub have many similarities. I think in your case this is not a q's of either or and should be any one of the above. - Raunak Jhawar

1 Answers

1
votes

Confluent partners with GCP, if you want to run a managed Kafka on GCP, you could use Confluent Cloud.

And yes, you can use PubSub and Kafka via Kafka Connect - https://github.com/GoogleCloudPlatform/pubsub/tree/master/kafka-connector

If you want a bunch of disparate microservices, it is a good idea to invest in learning a service mesh, and you have several options that are laid out at https://layer5.io/ , and Kubernetes can widely help manage these services, if you so chose.

Spring Cloud also has a service discovery component, based on Netflix Eureka - https://spring.io/guides/gs/service-registration-and-discovery/ plus Spring Cloud Netflix is a bunch of Netflix tools that are useful for microservices (Netflix is a large Kafka user, if you didn't know)