How do you get logstash to output to elasticsearch on a private ip?

2
votes

Unfortunately, Logstash is not attempting to output to Elasticsearch at the correct IP address. Elasticsearch is configured at a private ip address on port 9200. Logstash is attempting to output to Elasticsearch at localhost:9200. This is shown in the log below.

{:timestamp=>"2016-02-08T16:27:58.572000-0500", :message=>"Attempted to send a bulk request to Elasticsearch configured at '[\"http://localhost:9200/\"]', but Elasticsearch appears to be unreachable or down!", :client_config=>{:hosts=>["http://localhost:9200/"], :ssl=>nil, :transport_options=>{:socket_timeout=>0, :request_timeout=>0, :proxy=>nil, :ssl=>{}}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::Manticore, :logger=>nil, :tracer=>nil, :reload_connections=>false, :retry_on_failure=>false, :reload_on_failure=>false, :randomize_hosts=>false}, :error_message=>"Connection refused", :class=>"Manticore::SocketException", :level=>:error}

My configuration files are below: /etc/elasticsearch/elasticsearch.yml

network.host: PRIVATE_IP_ADDRESS

/opt/logstash/conf.d/logstash.conf

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["PRIVATE_IP_ADDRESS:9200"]
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

I have the same file at /etc/logstash/conf.d/logstash.conf because I didn't know where to put the logstash configuration file.

When I run curl PRIVATE_IP_ADDRESS:9200, I get the following output

{
  "name" : "Gabriel Summers",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.1.1",
    "build_hash" : "40e2c53a6b6c2972b3d13846e450e66f4375bd71",
    "build_timestamp" : "2015-12-15T13:05:55Z",
    "build_snapshot" : false,
    "lucene_version" : "5.3.1"
  },
  "tagline" : "You Know, for Search"
}

How can I configure Logstash to output to the Elasticsearch at PRIVATE_IP_ADDRESS:9200 instead of localhost:92000?

4
curlelasticsearchconfigurationlogstashelastic-stack
Could you indicate how did you install logstash ( rpm, deb, tarball) and how do you start it ? BTW I strongly advise you to change the elasticsearch cluster name as any of your buddies, connected to the same LAN might join your cluster. – devlearn
I used rpm to install Logstash. Thank you for the advice. I'll do that now. – Gen Ohta

4 Answers

0
votes

I would double check that logstash is actually using that configuration file.

0
votes

Just use follwing snippet to you conf file and it will work.

output {
  elasticsearch {
     hosts => "PRIVATE_IP_ADDRESS:9200"
  }
}
0
votes

It turns out that when I was restarting logstash all of the processes weren't being terminated. My config files weren't the problem.

Here are the commands I ran to solve this: ps -ef | grep logstash sudo kill EACH_PID sudo kill -9 PID_THAT_WASNT_KILLED_BEFORE sudo /etc/init.d/logstash start

0
votes

If you use the RPM installation then your configuration should be located under /etc/logstash/conf.d/.

So try to move the file you mentioned ( opt/logstash/conf.d/logstash.conf ) into the /etc/logstash/conf.d directory then restart your logstash service and check it out.