Kubernetes MongoDB pods with NFS persistent volume provisioning

Question

I have a kubernetes cluster and I have set up an NFS server as persistent volume for a mongodb deployment.

And I have set the PeristentVolume and PersistentVolumeClaim as below:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    name: mynfs 
spec:
  storageClassName: manual 
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: <nfs-server-ip>
    path: "/srv/nfs/mydata" 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany 
  resources:
    requests:
      storage: 1Gi

Everything works fine but the only problem is, I can't run more than 1 mongodb pods because I get the following error.

{"t":{"$date":"2020-10-15T15:16:39.140+00:00"},"s":"E", "c":"STORAGE", "id":20557, "ctx":"initandlisten","msg":"DBException in initAndListen, terminating","attr":{"error":"DBPathInUse: Unable to lock the lock file: /data/db/mongod.lock (Resource temporarily unavailable). Another mongod instance is already running on the /data/db directory"}}

That pod is always in CrashLoopBackOff and restarts and again to the same status.

I think the problem here is the same volume path mentioned in the mongodb deployment is trying to access by the two pods at the same time and when one pod is already have the lock, other pod failed.

Here's the mongodb deployment yaml.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment    
  labels:
    name: mongodb
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels: 
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo
        ports:
        -  containerPort: 27017
        env: 
        - name: MONGO_INITDB_ROOT_USERNAME
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: mongo-username
        - name: MONGO_INITDB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mongodb-secret
              key: mongo-password
        volumeMounts:
        - name: data  
          mountPath: /data/db
      volumes: 
        - name: data
          persistentVolumeClaim: 
            claimName: task-pv-claim

can someone please help me fix this?

Thank you.

AFAIK you cannot have multiple instances of MongoDB pointing to the same path; each mongod needs to have exclusive access to its own data files. I think instead of deployment you should use statefulset with volumclaimtemplates, there is great answer about that. Could you check that and let me know if that answer your question? — Jakub
Hi, I tried some methods mentioned there, but still getting the same error. I asked another question today, can you have a look at that? https://stackguides.com/questions/64386094/kubernetes-statefulset-with-nfs-persistent-volume — Jananath Banuka
Hi @Jananath Banuka, I have made an answer from this comment as I think that's the answer for your question. I saw your another question with statefulset and I see there is already an answer provided, let me know if that solved your issue or you still need some help. — Jakub
I found some feedback elsewhere while troubleshooting this myself. The problem isn't that there are multiple issues here. It appears to be the underlying NFS and how it handles file locks. My configuration only has a single instance of Mongo and it was throwing this error. Moving the directory with the lock file to a local bind volume made the problem go away. docs.mongodb.com/manual/administration/production-notes/… — Geoff Sweet

Jakub Jakub · Accepted Answer · 2020-10-27T07:22:25

Issue

This log entry already tells you what is the issue

{"t":{"$date":"2020-10-15T15:16:39.140+00:00"},"s":"E", "c":"STORAGE", "id":20557, "ctx":"initandlisten","msg":"DBException in initAndListen, terminating","attr":{"error":"DBPathInUse: Unable to lock the lock file: /data/db/mongod.lock (Resource temporarily unavailable). Another mongod instance is already running on the /data/db directory"}}

All members access the same volume and data.

AFAIK you cannot have multiple instances of MongoDB pointing to the same path, each MongoDB instance needs to have exclusive access to its own data files.

Solution

You can run your application as StatefulSet with volumeClaimTemplate which ensures that each replica will mount its own volume. There is great answer about that.

Kubernetes MongoDB pods with NFS persistent volume provisioning

1 Answers

Issue

Solution