Unable to map the logstash with ElasticSearch on linux

Question

I just run the below command and it shows all the running images on docker

sudo docker ps

Output:

CONTAINER ID    IMAGE             COMMAND                   CREATED           STATUS           PORTS                    NAMES
e14ace6bd419    a962b6541416      "/bin/bash /usr/loca…"    23 hours ago      Up 22 hours      0.0.0.0:5601->5601/tcp   trusting_chatterjee
00e6822bb991    28259852697e      "/usr/local/bin/dock…"    23 hours ago      Up 23 hours      9200/tcp, 9300/tcp       friendly_roentgen

I just want to link the logstash into elastic search and tried to run the below command

Command:

 sudo docker run -d --rm -it -v /home/sabharanikumar/logstash.conf e95781358676

Output:

989e2a8f4d9fd972c4f2102d726a68877c989b546800899abbb8c382fb62f04c

logstash.conf:

input
{
  stdin{}
}
output
{
  elasticsearch{ hosts => ["localhost:9200"] } 
}

Logstash Log:

[2019-08-23T09:40:53,833][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@elasticsearch:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@elasticsearch:9200/][Manticore::ResolutionFailure] elasticsearch: Name or service not known"}

I ran the sudo docker ps but the logstash is being listed.

Is there anything I missed it? Is there anywhere I need to change the host value?

You need to run sudo docker ps -a and then you will see your container with an Exit status. Run docker logs <container_name> and post the result if you still need help with figuring it out, but most probably the log will tell you exactly what the problem is. — Mihai
Also "localhost:9200" doesn't look right. This configuration is read inside a container. For that container localhost is the container itself. I think you want to point to friendly_roentgen instead. Regarding the container names, I suggest you give them names when you start them instead of letting docker choose another random name each time you start your containers. — Mihai
Updated with latest finds. I just updated the command and observed that logstash is being up and running now. but still it throws some error. — ArrchanaMohan

Stefano Stefano · Accepted Answer · 2019-08-23T09:39:47

As @Mihai pointed out in the comments, "localhost:9200" is not correct. The logstash container will try to communicate with itself on the port 9200 but nothing is listening on there.

The second problem I see is that you're passing the logstash configuration file as parameter (or command) to docker. Moreover you should avoid using the --link option since it's deprecated and prefer the network generation.

In other words, the commands should look like this:

docker network create \
    --driver bridge \
    --subnet=172.100.0.0/16 \
    --gateway=172.100.0.1 \
    my_elk_net

and then starting both the docker containers using the --network=my_elk_net option. e.g.

docker run -d \
    --name elasticsearch \
    --name my_elk_net \
    <ElasticSearchIMAGEID>

docker run -d \
    --name logstash \
    --network my_elk_net \
    -v "/home/arrchana/logstash.conf:/usr/share/logstash/config/logstash.yml:ro"
    <LogstashIMAGEID>

Your elastic search should look like this now:

input
{
  stdin{}
}
output
{
  elasticsearch{ hosts => ["elasticsearch:9200"] } 
}

In alternative, you can use docker-compose and avoid the creation of the network. A very basic example of a docker-compose.yml, that should work for you, should look like this:

version: '3.7'

services:
  elasticsearch:
    image: image_image

  logstash:
    image: logstash_image
    volumes:
      - /home/arrchana/logstash.conf:/usr/share/logstash/config/logstash.yml:ro
    depends_on:
     - elasticsearch

Unable to map the logstash with ElasticSearch on linux

1 Answers