Failover for elasticsearch and logstash

Question

I was working to configure logstash and elasticsearch. In my logstash config file output section.

elasticsearch {
    codec => json_lines
    cluster => "firstEsearch_cluster"
    protocol => "http"
    host => "192.168.56.3"
    port => "9200"
}

If I do this, I'm able to communicate to the elasticsearch instance

But now I have multiple nodes for elasticsearch on various machines where I replicate data to recover from failure and machines are in the same network.

Now when my machine 192.168.56.3 failed and 192.168.56.4 and 192.168.56.5 were running other nodes of elasticsearch, wasn't able to send logs to elasticsearch cluster, since machine 192.168.56.3 was down. So

1. What should be the output configuration for logstash, so that I can still send the logs to elasticsearch cluster, when one of the machine goes down

When I tried to do this:

elasticsearch {
    codec => json_lines
    cluster => "firstEsearch_cluster"
    protocol => "http"
    #host => "192.168.56.3"
    #port => "9200"
}

logstash wasn't able to connect to elasticsearch instance, and wasn't able to send logs.

Answer 1

Deploy a load balancer with a DNS name and point your elasticsearch output to it. The load balancer will route the requests to the active elasticsearch nodes.

Answer 2

You can run setup a local ES instance on your indexers that acts as a client node. Configure the node with the following options

node.master: false
node.data: false

Then just point your elasticsearch output to localhost. This client node won't store any data and just act as a loadbalancer to the rest of your nodes.

Answer 3

Elasticsearch (Now called just elastic) recomends setting up an nginx proxy which does load balancing. Point your logstash servers to the proxy, and nginx will round robin between the nodes in the cluster.

https://www.elastic.co/blog/playing-http-tricks-nginx/

Here is a snippet of what the nginx config might look like. Point your logstash servers at port 8080.

events {
    worker_connections  1024;
}

http {

  upstream elasticsearch {
    server 127.0.0.1:9200;
    server 127.0.0.1:9201;
    server 127.0.0.1:9202;

    keepalive 15;
  }

  server {
    listen 8080;

    location / {
      proxy_pass http://elasticsearch;
      proxy_http_version 1.1;
      proxy_set_header Connection "Keep-Alive";
      proxy_set_header Proxy-Connection "Keep-Alive";
    }

  }

}