Logstash can not connect to Elastic search

0
votes
{:timestamp=>"2017-07-19T15:56:36.517000+0530", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}
{:timestamp=>"2017-07-19T15:56:37.761000+0530", :message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:37:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:79:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in `call_once'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `perform_request'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:257:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/sniffer.rb:32:in `hosts'", "org/jruby/ext/timeout/Timeout.java:147:in `timeout'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/sniffer.rb:31:in `hosts'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:79:in `reload_connections!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:72:in `sniff!'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in `start_sniffing!'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:60:in `start_sniffing!'", "org/jruby/RubyKernel.java:1479:in `loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:59:in `start_sniffing!'"], :level=>:error}
{:timestamp=>"2017-07-19T15:56:38.520000+0530", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200\"]', but Elasticsearch appears to be unreachable or down!", :error_message=>"Connection refused (Connection refused)", :class=>"Manticore::SocketException", :level=>:error}

Though Elastic search in running on port 127.0.0.1:9200 I do not understand from where logstash is taking this configuration I have not configured logstash to connect elastic search on localhost

in logstash.service

ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"

and in /etc/logstash I have logstash.yml 

path.config: /etc/logstash/conf.d

in /etc/logstash/conf.d

output {
  elasticsearch { hosts => ["10.2.0.10:9200"] 
  manage_template => false
  index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  document_type => "%{[@metadata][type]}"
  }
}
2
elasticsearchlogstashkibana
Do you have another config file in /etc/logstash/conf.d ? And can you start logstash with the --debug flag and paste the output in your question? – Val

2 Answers

0
votes

conf.d is your directory right, you need a file there something like myconf.conf and be in following format:

input {

}

filter {
 #can be empty 
}

output {

}

Once you apply all your changes you need to restart your logstash service, and it will apply your new changes. You can also control that in your LS settings logstash.yml file, if you need it to restart itself, once you apply a new change to any of the files under conf.d

You can also break up your conf files like 1_input.conf 2_filter.conf and 99_output.conf such that each will contain its own plugin i.e. input , filter and output.

0
votes
  • Start Elasticsearch.

  • Write a conf file for Logstash to connect and upload data into Elasticsearch.

    input {
    file {
           type => "csv"
           path => "path for csv."
           start_position => "beginning"
         }
   }
filter {
     csv {
         columns => ["Column1","Column2"]
         separator => ","
        }
     mutate {
        convert => {"Column1" => "float"}
        }
   }
output {
     elasticsearch {
     hosts => "http://localhost:9200"
        }
    stdout { codec => rubydebug}
}

  • host for elasticsearch can be configured in elasticsearch.yml file.

  • run the conf file (logstash.bat -f abc.conf)