3
votes

In our network, there are local nodes that are not always connected to center node and we cannot change this condition.

Our requirements are as follows;

  1. In local nodes; local logs will be stored and can be searched
  2. In center node; all collected logs will be stored and can be searched

Our current logstash setup is as follows;

Logstash collects logs in local nodes and ships using redis also store in local elasticsearch instances. In center node input from redis is again stored central elasticsearch.

Since local nodes are not always connected; logs are not transmitted to center and logs are not available at center node.

So we need additional settings to transmit logs that are recorded while node is offline. Neither Logstash nor Elasticsearch documentation includes that kind of syncronisation. Do you have any advices?

1

1 Answers

0
votes

You can run a redis on local node with different port for center node. That's mean you will start two redis. One for local use and one for center node!

The local nodes also ships the log to this redis. Redis will buffer all the logs it received. (PS: Logstash can output to multiple redis in the same configuration)

When the network is connected, the center node can read the logs event from the local node redis. So this can solved all synchronization problem.