I'm trying to set up a centralized logging solution that uses filebeat on EC2 to ship logs directly to a VPC AWS Elasticsearch cluster (simple configuration, no logstash). I do not require any internet accessibility to Elasticsearch.
My VPC is 172.31.0.0/16
My Access Control Policy is wide open (which is only to the VPC)
My security groups allow TCP 443 and 9200
On the EC2 instance, I'm able to get a response from the https endpoint
curl https://vpc-<my-es-cluster>.<region>.es.amazonaws.com
{
"name" : "<name>",
"cluster_name" : "<account-id>:<my-es-cluster>",
"cluster_uuid" : "<uuid>",
"version" : {
"number" : "6.2.2",
"build_hash" : "10b1edd",
"build_date" : "2018-02-28T15:42:08.616107Z",
"build_snapshot" : false,
"lucene_version" : "7.2.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
From my EC2 instance 9200 is unavailable, which seems like a security group issue, however the sg attached to my domain allows 443 and 9200 from 172.31.0.0/16. I also tried opening the attached sg up to all traffic and that did not resolve my issue.
I did some research and found that this issue may be due to not setting the network.host configuration on the ES cluster to bind to 0.0.0.0 (or the private IP): Connection refused error on Elastic Search
Has anyone else run into this issue? This is a basic use case: an EC2 within a VPC failing to have network access to a VPC-internal ES cluster over 9200 despite an open security group.
