Elasticsearch Nodes behind a load balancer

Question

I setup a cluster of elasticsearch nodes on AWS. The cluster contains 3 nodes. The nodes are behind an ELB(Elastic Load balancer). The ELB has a DNS name say "a.b.c.com".

I am able to successfully run things like:

curl -XGET 'http://a.b.c.com:9200/_cat/nodes'

This proves that permissions and routes are working properly.

When I try to connect my Java Application server to Elasticsearch it fails with the following error:

org.elasticsearch.transport.ReceiveTimeoutTransportException: [][inet[a.b.c.com/172.31.27.110:9300]][cluster:monitor/nodes/info] request_id [57] timed out after [10000ms]
        at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:366)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

This happens regardless whether I increase client.transport.ping_timeout.

The java code i use to connect looks as follows:

    Settings settings = ImmutableSettings.settingsBuilder().put("cluster.name", clusterName).put("client.transport.ping_timeout", 10000).build();
    client = new TransportClient(settings);
    TransportClient transportClient = (TransportClient)client;
    transportClient.addTransportAddress(new InetSocketTransportAddress(instance, esPort));

instance="a.b.c.com" esPort=9300(i tried 9200 as well)

Thx in advance for helping out.

I don't remember how i solved it but it does work now :-( . Are you having the same exception? — isaac.hazan
Now that i think about it , the code i am using is the exact same as above. I think i solved it by making sure that i have rules on port 9300. Since the curl above just proves that 9200 works, not 9300. — isaac.hazan

Unknown Unknown · Accepted Answer · 2016-09-13T09:07:12

An answer here for any poor soul searching in the future:

You've got your ES node running on 9300, and your security group open on that instance to allow connections from your app server.

But the ELB has its own security groups. So you need to do this:

Set the SG on the ELB to allow connections FROM your app server. Set the SG on the ES instance to allow connections FROM your ELB.

and also set the SG on your ES instances to allow traffic on 9300 from themselves - once they discover each other they need to be able to talk to themselves!

Elasticsearch Nodes behind a load balancer

1 Answers