Elasticsearch helm chart pod stuck in “Init:CrashLoopBackOff” state

Question

I am deploying the helm chart for Elastic-stack on a bare-metal k8s cluster here https://github.com/helm/charts/tree/master/stable/elastic-stack

This includes the helm chart for Elasticsearch here https://github.com/helm/charts/tree/master/stable/elasticsearch

The Elastic-stack chart calls the Elasticsearch with the default values in values.yaml, and I am not setting anything else.

After helm installing, I see the pods elastic-stack-elasticsearch-data-0 and elastic-stack-elasticsearch-master-0 are stuck in Init:CrashLoopBackOff (after repeating Init:Error for some time).

kubectl describe pod shows me that the problem is with the initContainer called chown. The code for this container is here https://github.com/helm/charts/blob/master/stable/elasticsearch/templates/data-statefulset.yaml#L79

The relevant output from describe pod is not very helpful:

State:          Waiting
  Reason:       CrashLoopBackOff
Last State:     Terminated
  Reason:       Error
  Exit Code:    1
  Started:      Thu, 24 Jan 2019 05:35:14 +0000
  Finished:     Thu, 24 Jan 2019 05:35:14 +0000
Ready:          False
Restart Count:  1
Environment:    <none>
Mounts:
  /usr/share/elasticsearch/data from data (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from elastic-stack-elasticsearch-data-token-kgtsm (ro)

I know that it is able to pull the image, since it pulled it for the previous initContainer.

I have a feeling it has to do with the Mounts. What exactly is /usr/share/elasticsearch/data from data (rw) doing? I created persistentvolumes called es-data-volume and es-master-volume and they have been claimed by data-elastic-stack-elasticsearch-data-0 and data-elastic-stack-elasticsearch-master-0. Is that line looking for a volume named data?

I don't know where to look to troubleshoot this problem. What could be some possible causes of this issue?

I didn't realize I could get init container logs. kubectl logs -n logging elastic-stack-elasticsearch-master-0 -c chown gives: "+ chown elasticsearch:elasticsearch /usr/share/elasticsearch/data chown: changing ownership of '/usr/share/elasticsearch/data': Operation not permitted" This is weird because I exec'd into a different pod created by the elasticsearch-stack helm chart and ran the chown command myself. Is there a different permission associated with this? I notice that the chart with this initcontainer has securityContext.runAsUser=0 — swagrov

Marcin Gorzynski Marcin Gorzynski · Accepted Answer · 2019-02-17T12:23:34

I had the same issue and this has fixed for me, I changed settings on my NFS server (sudo vim /etc/exports)

from:

/data/nfs/kubernetes 192.168.1.0/24(rw,sync,no_subtree_check)

to :

/data/nfs/kubernetes 192.168.1.0/24(rw,insecure,sync,no_subtree_check,no_root_squash)

from what I understood no_root_squash is the key

Hopefully that will solve it for you as well

Elasticsearch helm chart pod stuck in “Init:CrashLoopBackOff” state

1 Answers