Spring boot application pod fails to find the mongodb pod on Kubernetes cluster

Question

I have a Spring Boot Application backed by MongoDB. Both are deployed on a Kubernetes cluster on Azure. My Application throws "Caused by: java.net.UnknownHostException: mongo-dev-0 (pod): Name or service not known" when it tries to connect to MongoDB.

I am able to connect to the mongo-dev-0 pod and run queries on the MongoDB, so there is no issue with the Mongo itself and it looks like the Spring boot is able to connect to Mongo Service and discover the pod behind the service.

How do I ensure the pods are discoverable by my Spring Boot Application? How do I go about debugging this issue?

Any help is appreciated. Thanks in advance.

Here is my config:

---
apiVersion: v1
kind: Service
metadata:
  name: mongo-dev
  labels:
    name: mongo-dev
spec:
  ports:
  - port: 27017
    targetPort: 27017
  clusterIP: None
  selector:
    role: mongo-dev
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo-dev
spec:
  serviceName: "mongo-dev"
  replicas: 3
  template:
    metadata:
      labels:
        role: mongo-dev
        environment: dev
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: mongo-dev
          image: mongo:3.4
          command:
            - mongod
            - "--replSet"
            - rs0
            - "--smallfiles"
            - "--noprealloc"
            - "--auth"
            - "--bind_ip"
            - 0.0.0.0
          ports:
            - containerPort: 27017
          volumeMounts:
            - name: mongo-dev-persistent-storage
              mountPath: /data/db
        - name: mongo-sidecar
          image: cvallance/mongo-k8s-sidecar
          env:
            - name: MONGO_SIDECAR_POD_LABELS
              value: "role=mongo-dev,environment=dev"
            - name: KUBERNETES_MONGO_SERVICE_NAME
              value: "mongo-dev"
  volumeClaimTemplates:
  - metadata:
      name: mongo-dev-persistent-storage
      annotations:
        volume.beta.kubernetes.io/storage-class: "devdisk"
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Gi
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: devdisk
provisioner: kubernetes.io/azure-disk
parameters:
  skuName: Premium_LRS
  location: abc
  storageAccount: xyz

Did you find the problem? Can you share the solution? Thanks — Elier

Rachid Habel Rachid Habel · Accepted Answer · 2018-12-06T17:16:39

To be able to reach your mongodb pod via its service from your spring boot application, you have to start the mongodb pod and the corresponding service first, and then start your spring boot application pod (let's name it sb-pod).

You can enforce this order by using an initContainer in your sb-pod; to wait for the database service to be available before starting. Something like:

initContainers:
  - name: init-mongo-dev
    image: busybox
    command: ['sh', '-c', 'until nslookup mongo-dev; do echo waiting for mongo-dev; sleep 2; done;']

If you connect to your sb-pod using:

kubectl exec -it sb-pod bash

and type the env command, make sure you can see the environment variables

MONGO_DEV_SERVICE_HOST and MONGO_DEV_SERVICE_PORT

Spring boot application pod fails to find the mongodb pod on Kubernetes cluster

2 Answers