0
votes

I have a microservices architecture whose logs have to be sent to a remote Kafka topic. Next to it, the consumer of this topic will send the logs to an ELK stack (an other team)

I want to have a dedicated microservice (fwk-proxy-elasticsearch) whose responsability is to collec the logs from the others one and send them to the remote kafka topic.

what's the best protocol to dispatch all the logs aggregated from my microservices to the fwk-proxy-elasticsearch microservice ?

I want this pattern to not duplicate the security configuration of the remote kafka topic. I want to centralize it in a single place.

May I use vertx event bus for that ? or kafka is beter ? or someother tool ? May I use vertx to send message from jvm to jvm ?

Moreover, in a microservice architecture, is it a good pattern to centralize a use case in a dedicated microservice? (remote http connection for example) On my point of view, it allows business microservices to focus on a business issue and not to worry over the protocol that the result has to be sent.

Thanks!

1
Why do you need a specific service when Filebeat can collect local log files and send to both Kafka and elasticsearch? - OneCricketeer

1 Answers

0
votes

I believe you can use both Vert.x event bus and Kafka to propagate the logs, there are pros and cons on each approach.

While I understand the reasoning behind this decision, I would still consider a dedicated solution built for this purpose, like Fluentd, which is able to aggregate the logs and push them into multiple sources (including Kafka, via the dedicated plugin). I'm sure there are other similar solutions.

There are a couple of important benefits that I see if you use a dedicated solution, instead of building it yourself:

  • The level of configurability, which is definitely useful in the future (in a dedicated solution, you need to write code each time you want to build something new)
  • The number of destinations where you can export the logs
  • Support for a hybrid architecture - with a few config updates, you will be able to grab logs from non-JVM microservices