Visualising static log files on Kibana

Question

I have a couple of Apache Server logs that I want to parse into Kibana for visualisation. Right now, I have tried setting up Elasticsearch + Kibana + Filebeat to ingest these server logs. However, Filebeat seems to be taking up quite a lot of CPU usage.

These logs are static, ie old logs that I collected. Therefore, I do not need Filebeat active monitoring capabilities. In this case, are there any other alternative methods for me to import my Apache Server logs into Kibana only once?

Also, as a side note, what are the differences between filebeat and logstash? I read that logstash is able to parse more info out of the logs such as geo location. However, while trying out my method, it seems that filebeat's apache module is capable of doing so as well.

Answer 1

Filebeat will monitor the logs in the given directory for changes. You can control that by either moving the processed files to an unwatched directory, or setting the ignore_older option. Also look at the options that start with close_ for more optimizations in this area.

Filebeat is a lighter-weight local agent that can do some processing and ship the logs to logstash or elasticsearch. Some local processing (e.g. multiline and ignore) are very powerful.

Logstash can also be used as a shipper, but it's typically used to turn unstructured logs into structured data for insertion into elasticsearch. Processing options are basically unlimited through the use of built-in filters and you can drop into ruby to do really custom stuff.