logstash doesn't succeed to send data to foreign elastic search

Question

Here is the error from logstash.err :

Faraday::ConnectionFailed: End of file reached call at /opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/adapter/net_http.rb:44 build_response at /opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/rack_builder.rb:139 run_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.0/lib/faraday/connection.rb:377 perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/transport/http/faraday.rb:24 call at org/jruby/RubyProc.java:271 perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/transport/base.rb:187 perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/transport/http/faraday.rb:20 perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.1/lib/elasticsearch/transport/client.rb:102 perform_request at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.1/lib/elasticsearch/api/namespace/common.rb:21 get_template at /opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.1/lib/elasticsearch/api/actions/indices/get_template.rb:24 template_exists? at /opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:132 template_install at /opt/logstash/lib/logstash/outputs/elasticsearch/protocol.rb:21 register at /opt/logstash/lib/logstash/outputs/elasticsearch.rb:259 each at org/jruby/RubyArray.java:1613 outputworker at /opt/logstash/lib/logstash/pipeline.rb:220 start_outputs at /opt/logstash/lib/logstash/pipeline.rb:152

Here is my output config :

output {
        elasticsearch { 
            host => "X.X.X.X"
            port => "9300"
            protocol => "http"
            cluster => "elasticsearch_david"
        }   
    }

No connection issue, any idea ?

Further investigations with tcpdump give :

GET /_template/logstash HTTP/1.1
User-Agent: Faraday v0.9.0
Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
Accept: */*
Connection: close

John Petrone John Petrone · Accepted Answer · 2014-06-13T00:49:41

The issue is with mismatching port and protocol:

output {
        elasticsearch { 
            host => "X.X.X.X"
            port => "9300"
            protocol => "http"
            cluster => "elasticsearch_david"
        }   
    }

You have protocol set to "http" which would require port 9200 (the default port that ES uses for http requests) but have the port set to 9300 which is the port used for inter cluster communications, normally used with the "node" protocol.

Unfortunately the documentation is contradictory about the default for protocol:

protocol
Value can be any of: "node", "transport", "http"
There is no default value for this setting.
Choose the protocol used to talk to Elasticsearch.

The ‘node’ protocol will connect to the cluster as a normal Elasticsearch node (but will not store data). This allows you to use things like multicast discovery. If you use the node protocol, you must permit bidirectional communication on the port 9300 (or whichever port you have configured).

The ‘transport’ protocol will connect to the host you specify and will not show up as a ‘node’ in the Elasticsearch cluster. This is useful in situations where you cannot permit connections outbound from the Elasticsearch cluster to this Logstash server.

The ‘http’ protocol will use the Elasticsearch REST/HTTP interface to talk to elasticsearch.

All protocols will use bulk requests when talking to Elasticsearch.

The default protocol setting under java/jruby is “node”. The default protocol on non-java rubies is “http”

Your best bet is to set the protocol setting to one of "node", "http" or "transport" depending on what you want to do and let logstash set the appropriate port for you:

output {
    elasticsearch { 
    host => "X.X.X.X"
    protocol => "http"
    cluster => "elasticsearch_david"
}

see http://logstash.net/docs/1.4.1/outputs/elasticsearch#protocol

logstash doesn't succeed to send data to foreign elastic search

2 Answers