ELK (Elasticsearch, Logstash, Kibana) stack - Do I really need both Logstash and Filebeat configured?

1
votes

I would like to deploy ELK stack on-premise for our custom application. So, I referred to the official docs for installation guides, installed Elasticsearch cluster and Kibana. Then comes the question: the documentation says I can process the logs from any custom app if I would like to (if built-in modules are not suitable for me), and I should just configure Filebeat so it can harvest these logs as an input. But what should be an output for Filebeat? I've heard that Elasticsearch should get processed, structured logs (for example, in JSON format) as an input; but our application produces plain text logs (as it's Java app, logs can include stack traces and other mixed data), and they should be processed and structured first... Or shouldn't they?

So, here are my questions regarding this situation:

  1. Do I need to set Filebeat output as Logstash input to format and structure logs, and then set Logstash output as Elasticsearch input? Or I can forward logs from Filebeat straight to Elasticsearch?
  2. Do I really need Filebeat in this situation, or maybe Logstash can be configured to read log files by its own?
1
elasticsearchlogginglogstashfilebeat

1 Answers

1
votes

Filebeat and Logstash can both work either on their own or in concert together. If all you have to do is to tail your log files and send them to Elasticsearch, without performing any processing on them, then I'd say go for Filebeat as it's more lightweight than Logstash.

If you need to perform some processing and transformation on your log files, then you have a few options depending on which solution you pick. You can leverage:

As a side note, I draw your attention on the fact that your Java app doesn't necessarily have to produce plain text logs. Using ecs-logging-java, it can also produce JSON logs ready to be ingested into Elasticsearch.

If you use the above logging library, then Filebeat would be perfectly suitable for your use case, but it depends of course on whether you need to parse and process the message field in your logs or not.