Best approach to send Scribe centralized logging output to elasticsearch

Question

We're currently building a centralized logging platform for our multitude of services. The plan is to install a scribe client on every application server, which will forward the logs to a central scribe server.

The central scribe server is currently logging all the aggregated logs to disk. But now we want to index them on ElasticSearch for visualization.

My exact question is: What's the best (most robust, fault-tolerant, less computationally expensive & memory efficient) way to forward logs from a central scribe server's output to ElasticSearch for indexing?

Few ideas worth exploring:

1. Scribe Server -> File -> Logstash -> ElasticSearch
2. Scribe Server -> File -> [X] -> RabbitMQ -> Logstash -> ElasticSearch
3. Scribe Server -> File -> [Y] -> ElasticSearch
4. Scriber Server -> [Z] -> Elastic Search

Any good options for X, Y or Z? Which one's the best approach? Please suggest.

Thanks.

Answer 1

According to me Z option is the best approach. ElasticSearch is suitable for fault tolerant system. There is shard and replica definition for this mechanism. But you must consider about memory efficiency . The count of your indexed document will be increased by the time and it will require at least 2 times more storage space as there is only one shard and one replica definition in the default config file.

EDIT:
If you have already a central scribe server I assume that you have all log messages on your central server. But all the options can be used for your app if you have no messaging structure for your central server. If you want to achieve your indexing asynchronously, you can use RabbitMQ or Apache solition. I think you can have a look at elastic search performance considerations for indexing. What is your indexing count per second ? Is it vital for your app? My other suggestion is MongoDB. If you don't insist on ElasticSearch , I could suggest you to search on MongoDB. You should check this comparison.