ElasticSearch Unable to revive connection: http://elasticsearch:9200/

Question

I tried to run ELK on Centos 8 with docker-compose :

here my docker-compose.yml

version: '3.1'

services:

  elasticsearch:
   image: docker.elastic.co/elasticsearch/elasticsearch:6.2.4
   container_name: elasticsearch
   hostname: elasticsearch
   ports:
    - "9200:9200"
   expose:
    - "9200"
   volumes:
    - elasticsearch-data:/usr/share/elasticsearch/data
   networks:
    - docker-network

  kibana:
   image: docker.elastic.co/kibana/kibana:6.2.4
   container_name: kibana
   ports:
    - "5601:5601"
   expose:
    - "5601"
   environment:
    - SERVER_NAME=kibana.localhost
    - ELASTICSEARCH_URL=http://elasticsearch:9200
    - ELASTICSEARCH_USERNAME=elastic
    - ELASTICSEARCH_HOST=elasticsearch
    - ELASTICSEARCH_PORT=9200
    - ELASTIC_PWD=changeme
    - KIBANA_PWD=changeme
   depends_on:
    - elasticsearch
   networks:
    - docker-network

networks:
  docker-network:
    driver: bridge

volumes:
  elasticsearch-data:

but i'm facing with this error :

{"type":"log","@timestamp":"2020-03-03T22:53:19Z","tags":["warning","elasticsearch","admin"],"pid":1,"message":"Unable to revive connection: http://elasticsearch:9200/"}

while i checked:

elasticsearch is running fine.
docker exec kibana ping elasticsearch work fine.
both kibana and elasticsearch are on same network as you can see in docker-compose.yml
i checked docker exec kibana curl http://elasticsearch:9200 and result is :

Failed connect to elasticsearch:9200; No route to host

I also check other similar problems and their solution but none of them worked.

Your ES could be starting and stoping repeatedly (not caught by ping). Check your Es logs or log into the container itself and check the process (ps -aux). Also, no need for expose: and depends-on:, they're archaic (you can't rely on it to reconnect when it drops at run time). The ports: is also not needed unless you want to access ES outside the host. I presume you would rather connect to Kibana instead from outside. This also keep the ES services hidden (secure) from direct outside connection. — Bernard
@Bernard I checked elasticsearch is running fine. i checked both logs and testing localhost:9200 — Ali Zeinali

James McGuigan James McGuigan · Accepted Answer · 2020-07-09T15:29:35

If you are running ElasticSearch inside Docker, then you may need to check if you have allocated sufficient memory limits to Docker. This can cause ElasticSearch to slowdown and even crash.

By default Docker Desktop is set to allow 2Gb of RAM per Docker, but in my own project I found that 4Gb prevented crashing, but 5Gb produced an additional performance speedup. Your mileage may vary depending on the amount of data you are ingesting.

Docker Desktop memory settings can be set via:

Docker Desktop -> Preferences -> Resources -> Memory

To inspect memory usage within the Docker container

DOCKER_ID=`docker ps | tail -n1 | awk '{ print $1 }'`; docker exec -it $DOCKER_ID /bin/bash

free -h  # repeatedly run to inspect changes over time

Note that ElasticSearch memory usage peaks during ingest and indexing and then eventually settle down to a slightly lower number once indexing and consolidation is complete. So ideally peak memory usage should be tested during ingest.

ElasticSearch Unable to revive connection: http://elasticsearch:9200/

1 Answers