Elasticsearch service running on minikube cluster not reachable from within the cluster

Question

I am using kompose to deploy this docker-compose.yaml

version: '3'
services:
  webapp:
    build:
      context: ../../../
      dockerfile: config/docker/dev/Dockerfile-dev
    container_name: myWebApp-dev
    command: ["/bin/sh", "-ec","sleep 1000"]
    image: 'localhost:5002/webapp:1'
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
      - ELASTICSEARCH_HOST=elasticsearch
    labels:
      kompose.image-pull-policy: 'IfNotPresent'
      kompose.service.type: nodeport
    ports:
      - "4000:4000"
      - "3000:3000"
    depends_on:
      - elasticsearch
    links:
      - elasticsearch

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.0.1
    container_name: elasticsearch
    command: ["/bin/sh", "-ec","sleep 1000"]
    environment:
      - node.name=elasticsearch
      - discovery.seed_hosts=es02
      - cluster.initial_master_nodes=elasticsearch,es02
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9300:9300

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.0.1
    container_name: es02
    command: ["/bin/sh", "-ec","sleep 1000"]
    environment:
      - node.name=es02
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=elasticsearch,es02
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata02:/usr/share/elasticsearch/data

to minikube. The elasticsearch pod and service are running. However, the webapp cannot access the elasticsearch cluster as I get a connection refused error when curling from within the webapp pod -> curl: (7) Failed to connect to 10.108.5.31 port 9200: Connection refused. Does anyone know what the reason for this problem is and how to fix it?

Abdullah Al Maruf - Tuhin Abdullah Al Maruf - Tuhin · Accepted Answer · 2019-06-13T04:37:06

In elasticsearch section, you have a shell command to sleep. And, never started any elasticsearch instances after that.

command: ["/bin/sh", "-ec","sleep 1000"]

So, looks like, there is no elasticsearch running inside the container and that's why connection refused is happening.

To Fix:

Get rid of command: of elasticsearch and es02, that way, default command will be used.

Note:

Now, When the elasticsearch starts, You will face two error (described below) with this compose yaml in kubernetes. These are unrelated to this post, But I will try to giving you direction where to look.

ERROR: [2] bootstrap checks failed
[1]: memory locking requested for elasticsearch process but memory is not locked
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

Here,

you need to update host system for vm.max_map_count. Exec into minikube virtualbox by minikube ssh and run sudo -s sysctl -w vm.max_map_count=262144 to change the map_count of host kernel. It will work, because docker/container doesn't provide kernel level isolation.

For minikube,

minikube ssh 'sudo -s sysctl -w vm.max_map_count=262144'

ulimit is not available in kompose. See issue here. So either you have to get rid of both, bootstrap.memory_lock=true from environment: sections, or you may need to update the docker image. This question is already asked here in stackoverflow.

So the improved kompose yaml (works well on minikube):

version: '3'
services:
  webapp:
    build:
      context: ../../../
      dockerfile: config/docker/dev/Dockerfile-dev
    container_name: myWebApp-dev
    command: ["/bin/sh", "-ec","sleep 1000"]
    image: 'localhost:5002/webapp:1'
    environment:
      - ELASTICSEARCH_URL=http://elasticsearch:9200
      - ELASTICSEARCH_HOST=elasticsearch
    labels:
      kompose.image-pull-policy: 'IfNotPresent'
      kompose.service.type: nodeport
    ports:
      - "4000:4000"
      - "3000:3000"
    depends_on:
      - elasticsearch
    links:
      - elasticsearch

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.0.1
    container_name: elasticsearch
    environment:
      - node.name=elasticsearch
      - discovery.seed_hosts=es02
      - cluster.initial_master_nodes=elasticsearch,es02
      - cluster.name=docker-cluster
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - esdata01:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9300:9300

  es02:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.0.1
    container_name: es02
    environment:
      - node.name=es02
      - discovery.seed_hosts=elasticsearch
      - cluster.initial_master_nodes=elasticsearch,es02
      - cluster.name=docker-cluster
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    volumes:
      - esdata02:/usr/share/elasticsearch/data

However, I would suggest to follow the elasticsearch official doc instead of using compose to install elasticsearch in kubernetes.

Elasticsearch service running on minikube cluster not reachable from within the cluster

1 Answers