Kubernetes PersistentVolumeClaim issues in AWS

Question

We have success creating the pods, services and replication controllers according to our project requirements. Now we are planning to setup persistence storage in AWS using Kubernetes. I have created the YAML file to create an EBS volume in AWS, it's working fine as expected. I am able to claim volume and successfully mount to my pod (this is for single replica only).

But when I am trying to create more the one replica, my pods are not creating successfully. When I am trying to create volumes, it's creating in only one availability zone. If my pod is created in a different zone node, since my volume is already created in different zone, due to that my pod is not creating successfully. How to create volumes in different zones for same application? How to make it successful, along with replica? How to create my persistent volumes claims?

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mongo-pvc
  labels:
    type: amazonEBS
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: ReplicationController
metadata:
  labels:
    name: mongo-pp
  name: mongo-controller-pp
spec:
  replicas: 2
  template:
    metadata:
      labels:
        name: mongo-pp
    spec:
      containers:
      - image: mongo
        name: mongo-pp
        ports:
        - name: mongo-pp
          containerPort: 27017
          hostPort: 27017
        volumeMounts:
        - mountPath: "/opt/couchbase/var"
          name: mypd1
      volumes:
      - name: mypd1
        persistentVolumeClaim:
          claimName: mongo-pvc

Note: Kubernetes 1.12 (Sept. 2018) introduces Topology aware dynamic provisioning, which might help: stackoverflow.com/a/52570486/6309 — VonC

Radek 'Goblin' Pieczonka Radek 'Goblin' Pieczonka · Accepted Answer · 2018-01-26T08:37:39

When you are using ReadWriteOnce volumes (ones that can not be mounted to multiple pods at the same time), simple PV/PVC creation will not cut it.

Both PV and PVC are pretty "singular" in a way that if you refer in Deployment to a particular claim name, your pods will all try to get the same one claim and the same one pv bound to that claim, resulting in a race condition where only one of the pods will be the first and only allowed to mount that RWO storage.

To mitigate this, you should use not PVC directly but via volumeClaimTemplates that will create PVC dynamicaly for every new pod scaled, like below :

 volumeClaimTemplates:
  - metadata:
      name: claimname
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

Kubernetes PersistentVolumeClaim issues in AWS

2 Answers