Running elasticsearch on Google Cloud Kubernetes ends in CrashLoopBackOff

Question

I try to run the elasticsearch6 container on a google cloud instance. Unfortunately the container always ends in CrashLoopBackOff. This is what I did:

install gcloud and kubectl

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb http://packages.cloud.google.com/apt cloud-sdk-$(lsb_release -c -s) main" | sudo tee -a /etc/apt/sources.list.d/google-cloud-sdk.list
sudo apt-get update && sudo apt-get install google-cloud-sdk kubectl

configure gcloud

gcloud init
gcloud config set compute/zone europe-west3-a  # For Frankfurt

create kubernetes cluster

gcloud container clusters create elasticsearch-cluster --machine-type=f1-micro --num-nodes=3

Activate pod

kubectl create -f pod.yml  

apiVersion: v1
kind: Pod
metadata:
  name: test-elasticsearch
  labels:
    name: test-elasticsearch
spec:
  containers:
    - image: launcher.gcr.io/google/elasticsearch6
      name: elasticsearch

After this I get the status:

kubectl get pods
NAME                    READY     STATUS             RESTARTS   AGE
test-elasticsearch   0/1       CrashLoopBackOff   10         31m

A kubectl logs test-elasticsearch does not show any output.

And here the output of kubectl describe po test-elasticsearch with some info XXX out.

Name:         test-elasticsearch
Namespace:    default
Node:         gke-elasticsearch-cluste-default-pool-XXXXXXXX-wtbv/XX.XXX.X.X
Start Time:   Sat, 12 May 2018 14:54:36 +0200
Labels:       name=test-elasticsearch
Annotations:  kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container elasticsearch
Status:       Running
IP:           XX.XX.X.X
Containers:
  elasticsearch:
    Container ID:   docker://bb9d093df792df072a762973066d504a4e7d73b0e87d0236a94c3e8b972d9c41
    Image:          launcher.gcr.io/google/elasticsearch6
    Image ID:       docker-pullable://launcher.gcr.io/google/elasticsearch6@sha256:1ddafd5293dbec8fb73eabffa29614916e4933bb057db50231084d89f4a0b3fa
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Sat, 12 May 2018 14:55:06 +0200
      Finished:     Sat, 12 May 2018 14:55:09 +0200
    Ready:          False
    Restart Count:  2
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-XXXXX (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  default-token-XXXXX:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-XXXXX
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.alpha.kubernetes.io/notReady:NoExecute for 300s
                 node.alpha.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                From                                                          Message
  ----     ------                 ----               ----                                                          -------
  Normal   Scheduled              51s                default-scheduler                                             Successfully assigned test-elasticsearch to gke-elasticsearch-cluste-def
  Normal   SuccessfulMountVolume  51s                kubelet, gke-elasticsearch-cluste-default-pool-XXXXXXXX-wtbv  MountVolume.SetUp succeeded for volume "default-token-XXXXX"
  Normal   Pulling                22s (x3 over 49s)  kubelet, gke-elasticsearch-cluste-default-pool-XXXXXXXX-wtbv  pulling image "launcher.gcr.io/google/elasticsearch6"
  Normal   Pulled                 22s (x3 over 49s)  kubelet, gke-elasticsearch-cluste-default-pool-XXXXXXXX-wtbv  Successfully pulled image "launcher.gcr.io/google/elasticsearch6"
  Normal   Created                22s (x3 over 48s)  kubelet, gke-elasticsearch-cluste-default-pool-XXXXXXXX-wtbv  Created container
  Normal   Started                21s (x3 over 48s)  kubelet, gke-elasticsearch-cluste-default-pool-XXXXXXXX-wtbv  Started container
  Warning  BackOff                4s (x3 over 36s)   kubelet, gke-elasticsearch-cluste-default-pool-XXXXXXXX-wtbv  Back-off restarting failed container
  Warning  FailedSync             4s (x3 over 36s)   kubelet, gke-elasticsearch-cluste-default-pool-XXXXXXXX-wtbv  Error syncing pod

Ok first of all, you should not create a pod manually this way, but always relay on some kind of controller (a Deployment, ReplicationController, StatefulSet, etc). That being said, try running kubectl describe po test-elasticsearch and post the output — whites11
@whites11 after getting this working I would love to get a link to a good documentation how to properly set this up! — Fabian
Looks like a runtime error, are you sure kubectl logs returns nothing? — whites11
Unfortunately yes, no output. Running the same container on my local docker works :-( Hmm maybe it has something to do with vm.max_map_count=262144 which is mentioned on the elasticsearch docu. But I am not sure how to set it on GKE. — Fabian
I see the exit status is 137 which usually means out of memory. How much memory do you have in your node(s)? — whites11

Fabian Fabian · Accepted Answer · 2018-05-12T14:03:50

The problem was the f1-micro instance. It doesn't have enough memory to run. Only after upgrading to an instance with 4GB it works. Unfortunately this is way too expensive for me, so I have to look for something else.