What's the difference of using nfs/efs as PersistentVolume and as a normal volume in kubernetes/k8s StatefulSet?

Question

I was trying to install a Cassandra stateful set on our k8s cluster. We want to mount an AWS EFS to all the Cassandra pods. And during configuring the volumes, I found we have 2 ways of declaring the volumes.

Create an EFS, install efs-provisioner (a persistentVolume provisioner plugin) on our cluster, create a PersistentVolume with the EFS-provisioner, and use volumeClaimTemplates (I think it actually means persistentVolumeClaimTemplates, I don' know why they ignore the persistent prefix) in pod templates. Like this :

apiVersion: v1
kind: Pod
metadata:
  name: hello-openshift-nfs-pod 
  labels:
    name: hello-openshift-nfs-pod
spec:
  containers:
    - name: hello-openshift-nfs-pod
      image: openshift/hello-openshift 
      ports:
        - name: web
          containerPort: 80
      volumeMounts:
        - name: nfsvol 
          mountPath: /usr/share/nginx/html 
  securityContext:
      supplementalGroups: [100003] 
      privileged: false
  volumes:
    - name: nfsvol
      persistentVolumeClaim:
        claimName: nfs-pvc

(code from a article)

Or I can mount the EFS directly to my Cassandra pods :

apiVersion: v1
kind: Pod
metadata:
  name: hello-openshift-nfs-pod 
  labels:
    name: hello-openshift-nfs-pod
spec:
  containers:
    - name: hello-openshift-nfs-pod
      image: openshift/hello-openshift 
      ports:
        - name: web
          containerPort: 80
      volumeMounts:
        - name: nfsvol 
          mountPath: /usr/share/nginx/html 
  securityContext:
      supplementalGroups: [100003] 
      privileged: false
  volumes:
    - name: cassandra-shared-volume
      nfs:
        server: <EFS file system endpoint url>
        path: "/cassandra/shared"

So for me, method 1 is just like wrap the nfs(EFS in this case) in a persistentVolume. I don't know what is the major benefits of method 1 though it is almost how every team use nfs. There is a short answer on efs-provisioner FAQ : they says

I noticed the EFS gets mounted directly to the efs-provisioner container, can I do that for my apps? Yes you can but it's not recommended. You lose the reusability of the StorageClass to be able to dynamically provision new PersistentVolumes for new containers and pods.

This answer is fine but I am wondering if there is any implicts behind persistentVolume than volume alone. There is StatefulSet doc saying :

The storage for a given Pod must either be provisioned by a PersistentVolume Provisioner based on the requested storage class, or pre-provisioned by an admin.

So I am wondering whether s StatefulSet must have a storage type of: PersistentVolume. The major conflicts is , unlike block storage, EFS itself is persistent and it's dynamic scaling ability is already handled by AWS and doesn't need persistentVolumeProvisioner to create new resources.

I want to know is there any major benefit of method 1 or flaw in method 2 that I must use method 1?

Carlos Monroy Nieblas Carlos Monroy Nieblas · Accepted Answer · 2019-09-13T04:09:40

Use of SAN and network filesystems like EFS is not recommended for Cassandra in general, due to performance concerns. Depending on the workflows, you can get acceptable performance with ephemeral (instance) storage vs EBS, as explained here.

Adding the containers layer has not prevented those issues, at this moment we're conducting a similar Proof of Concept setting a write-intensive scenario, and we have been able to get some acceptable performance using i3 instances and ephemeral storage for the PersistentVolumes definition, but this is not a definitive response yet as we continue looking for the right configuration for our purposes.

What's the difference of using nfs/efs as PersistentVolume and as a normal volume in kubernetes/k8s StatefulSet?

1 Answers