Difference between using Filebeat and Logstash to push log file to Elasticsearch

Question

I am trying out the ELK to visualise my log file. I have tried different setups:

1. Logstash file input plugin https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html
2. Logstash Beats input plugin https://www.elastic.co/guide/en/logstash/current/plugins-inputs-beats.html with Filebeat Logstash output https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html
3. Filebeat Elasticsearch output https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html

Can someone list out their differences and when to use which setup? If it is not for here, please point me to the right place like Super User or DevOp or Server Fault.

Answer 1

1) To use logstash file input you need a logstash instance running on the machine from where you want to collect the logs, if the logs are on the same machine that you are already running logstash this is not a problem, but if the logs are on remote machines, a logstash instance is not always recommended because it needs more resources than filebeat.

2 and 3) For collecting logs on remote machines filebeat is recommended since it needs less resources than a logstash instance, you would use the logstash output if you want to parse your logs, add or remove fields or make some enrichment on your data, if you don't need to do anything like that you can use the elasticsearch output and send the data directly to elasticsearch.

This is the main difference, if your logs are on the same machine that you are running logstash, you can use the file input, if you need to collect logs from remote machines, you can use filebeat and send it to logstash if you want to make transformations on your data, or send directly to elasticsearch if you don't need to make transformations on your data.

Another advantage of using filebeat, even on the logstash machine, is that if your logstash instance is down, you won't lose any logs, filebeat will resend the events, using the file input you can lose events in some cases.

Answer 2

Not really familiar with (2). But, Logstash(1) is usually a good choice to take a content play around with it using input/output filters, match it to your analyzers, then send it to Elasticsearch. Ex. You point the Logstash to your MySql which takes a row modify the data (maybe do some math on it, then Concat some and cut out some words then send it to ElasticSearch as processed data).

As for Logbeat(2), it's a perfect choice to pick up an already processed data and pass it to elasticsearch. Logstash (as the name clearly states) is mostly good for log files and stuff like that. usually you can do tiny changes to those. Ex. I have some log files in my servers (incl errors, syslogs, process logs..) Logstash listens to those files, automatically picks up new lines added to it and sends those to Elasticsearch. Then you can filter some things in elasticsearch and find what's important to you.

p.s: logstash has a really good way of load balancing too many data to ES.