How to create an kubernetes NFS volume on Google Container Engine

Question

I am trying to create a kubernetes NFS volume on Google Container Engine (GKE) and get it used by a deployment.

I did this in several steps as it shown in this github repository kubernetes-nfs-volume-on-gke:

Create a GKE cluster and GCE persistent disk
Config the context for the kubectl to deal with the GKE cluster
Creation of the PersistentVolume (PV) and the PersistentVolumeClaim (PVC)
Creation of an NFS server
Create a service for the NFS server to expose it (the IP address of that service is used for the creation of the NFS PV and NFS PVC)
Creation of NFS volume
Create a Deployment of a busybox for checking the NFS volume is accessible.

After fellowing these step, this is the obtained error:

$ kubectl describe pods  nfs-busybox-2762569073-lhb5p
Name:       nfs-busybox-2762569073-lhb5p
Namespace:  default
Node:       gke-mappedinn-cluster-default-pool-f94cb0d4-fmfb/10.240.0.3
Start Time: Wed, 12 Apr 2017 04:12:20 +0400
Labels:     name=nfs-busybox
        pod-template-hash=2762569073
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"nfs-busybox-2762569073","uid":"b1e523ae-1f14-11e7-a084-42010a8e0...
        kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container busybox
Status:     Pending
IP:     
Controllers:    ReplicaSet/nfs-busybox-2762569073
Containers:
  busybox:
    Container ID:
    Image:      busybox
    Image ID:       
    Port:       
    Command:
      sh
      -c
      while true; do date > /mnt/index.html; hostname >> /mnt/index.html; sleep $(($RANDOM % 5 + 5)); done
    State:      Waiting
      Reason:       ContainerCreating
    Ready:      False
    Restart Count:  0
    Requests:
      cpu:      100m
    Environment:    <none>
    Mounts:
      /mnt from my-pvc-nfs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-20n4b (ro)
Conditions:
  Type      Status
  Initialized   True
  Ready     False
  PodScheduled  True
Volumes:
  my-pvc-nfs:
    Type:   PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  nfs
    ReadOnly:   false
  default-token-20n4b:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-20n4b
    Optional:   false
QoS Class:  Burstable
Node-Selectors: <none>
Tolerations:    <none>
Events:
  FirstSeen LastSeen    Count   From                                SubObjectPath   Type        Reason      Message
  --------- --------    -----   ----                                -------------   --------    ------      -------
  5m        5m      1   default-scheduler                               Normal      Scheduled   Successfully assigned nfs-busybox-2762569073-lhb5p to gke-mappedinn-cluster-default-pool-f94cb0d4-fmfb
  3m        48s     2   kubelet, gke-mappedinn-cluster-default-pool-f94cb0d4-fmfb           Warning     FailedMount Unable to mount volumes for pod "nfs-busybox-2762569073-lhb5p_default(b1e7c901-1f14-11e7-a084-42010a8e0116)": timeout expired waiting for volumes to attach/mount for pod "default"/"nfs-busybox-2762569073-lhb5p". list of unattached/unmounted volumes=[my-pvc-nfs]
  3m        48s     2   kubelet, gke-mappedinn-cluster-default-pool-f94cb0d4-fmfb           Warning     FailedSync  Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"nfs-busybox-2762569073-lhb5p". list of unattached/unmounted volumes=[my-pvc-nfs]
  37s       37s     1   kubelet, gke-mappedinn-cluster-default-pool-f94cb0d4-fmfb           Warning     FailedMount MountVolume.SetUp failed for volume "kubernetes.io/nfs/b1e7c901-1f14-11e7-a084-42010a8e0116-nfs" (spec.Name: "nfs") pod "b1e7c901-1f14-11e7-a084-42010a8e0116" (UID: "b1e7c901-1f14-11e7-a084-42010a8e0116") with: mount failed: exit status 32
Mounting command: /home/kubernetes/bin/mounter
Mounting arguments: 10.247.250.208:/exports /var/lib/kubelet/pods/b1e7c901-1f14-11e7-a084-42010a8e0116/volumes/kubernetes.io~nfs/nfs nfs []
Output: Running mount using a rkt fly container
run: group "rkt" not found, will use default gid when rendering images

In the kubernetes dashboard, the error is as follows:

Unable to mount volumes for pod "nfs-busybox-2762569073-lhb5p_default(b1e7c901-1f14-11e7-a084-42010a8e0116)": timeout expired waiting for volumes to attach/mount for pod "default"/"nfs-busybox-2762569073-lhb5p". list of unattached/unmounted volumes=[my-pvc-nfs]

Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"nfs-busybox-2762569073-lhb5p". list of unattached/unmounted volumes=[my-pvc-nfs]

Have I missed something?

Thanks,

I am planning to write a NFS on GKE tutorial nowadays. I don’t know the answer right away, but thanks for asking this question as it validates the need for such a tutorial. If you want to stay connected, my email is ahmetb✳︎google.com. I can answer this in a few weeks. — Ahmet Alp Balkan
Thanks for your proposal... I am still working on... If I find a solution, I will post it here... If you want to make a PR kubernetes-nfs-volume-on-gke it will be great! — Amine Jallouli

surajd surajd · Accepted Answer · 2017-04-20T06:37:34

This comment in the issue on kubernetes seems to solve this NFS issue on GKE.

Qutoing that comment:

Edit examples/volumes/nfs/nfs-pv.yaml change the last line to path: "/".

Edit examples/volumes/nfs/nfs-server-rc.yaml change the image to the one that enabled NFSv4 image: gcr.io/google_containers/volume-nfs:0.8

Also there are other issues where this is tracked here and here.

How to create an kubernetes NFS volume on Google Container Engine

1 Answers