In my Kubernetes cluster, Rancher never creates Persistent Volumes after creating a Persistent Volume Claim and applying a Pod.
Solution/Work around available under second update.
The cluster has been installed with Kubespray. The configuration for local path provisioning in inventory/myclster/group_vars/k8s-cluster/addons.yml
:
# Rancher Local Path Provisioner
local_path_provisioner_enabled: true
# local_path_provisioner_namespace: "local-path-storage"
# local_path_provisioner_storage_class: "local-path"
# local_path_provisioner_reclaim_policy: Delete
# local_path_provisioner_claim_root: /opt/local-path-provisioner/
# local_path_provisioner_debug: false
# local_path_provisioner_image_repo: "rancher/local-path-provisioner"
# local_path_provisioner_image_tag: "v0.0.14"
# local_path_provisioner_helper_image_repo: "busybox"
# local_path_provisioner_helper_image_tag: "latest"
# Local volume provisioner deployment
local_volume_provisioner_enabled: false
# local_volume_provisioner_namespace: kube-system
# local_volume_provisioner_nodelabels:
# - kubernetes.io/hostname
# - topology.kubernetes.io/region
# - topology.kubernetes.io/zone
# local_volume_provisioner_storage_classes:
# local-storage:
# host_dir: /mnt/disks
# mount_dir: /mnt/disks
# volume_mode: Filesystem
# fs_type: ext4
# fast-disks:
# host_dir: /mnt/fast-disks
# mount_dir: /mnt/fast-disks
# block_cleaner_command:
# - "/scripts/shred.sh"
# - "2"
# volume_mode: Filesystem
# fs_type: ext4
Steps to recreate the problem:
Create PVC:kubectl create -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pvc.yaml
Result:
Created PVC, name local-path-pvc
, status Pending, storage class local-path
Create Podkubectl create -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/examples/pod.yaml
Result:
Created Pod, name create-pvc-123
, status Waiting:ContainerCreating.
Describing Pod with kubectl
:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling <unknown> error while running "VolumeBinding" prebind plugin for pod "create-pvc-123": Failed to bind volumes: timed out waiting for the condition
I have tried different charts, and in all cases no Persistent Volume has been created. The ServiceAccount local-path-provisioner-service-account exists. The Deployment of the local path provisioner has one Pod.
UPDATE
On the server the logs contains several errors, sudo journalctl -xeu kubelet | grep 'fail'
:
...
Oct 12 16:53:36 node1 kubelet[274306]: E1012 16:53:36.000246 274306 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/configmap/71b44438-fadb-4859-a788-8d911dfab2db-script podName:71b44438-fadb-4859-a788-8d911dfab2db nodeName:}" failed. No retries permitted until 2020-10-12 16:54:40.000164134 +0200 CEST m=+9380.643933974 (durationBeforeRetry 1m4s). Error: "MountVolume.SetUp failed for volume \"script\" (UniqueName: \"kubernetes.io/configmap/71b44438-fadb-4859-a788-8d911dfab2db-script\") pod \"create-pvc-80d115d9-98fd-4fcd-9e41-55b74f809efb\" (UID: \"71b44438-fadb-4859-a788-8d911dfab2db\") : configmap references non-existent config key: setup"
Oct 12 16:53:36 node1 kubelet[274306]: E1012 16:53:36.404015 274306 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script podName:424b196e-5132-479a-8b95-63e41e0ea124 nodeName:}" failed. No retries permitted until 2020-10-12 16:54:40.403980548 +0200 CEST m=+9381.047750378 (durationBeforeRetry 1m4s). Error: "MountVolume.SetUp failed for volume \"script\" (UniqueName: \"kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script\") pod \"create-pvc-3b132d90-8812-4391-bc29-966ee47bee0d\" (UID: \"424b196e-5132-479a-8b95-63e41e0ea124\") : configmap references non-existent config key: setup"
Oct 12 16:54:40 node1 kubelet[274306]: E1012 16:54:40.464999 274306 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script podName:424b196e-5132-479a-8b95-63e41e0ea124 nodeName:}" failed. No retries permitted until 2020-10-12 16:56:42.464936126 +0200 CEST m=+9503.108706016 (durationBeforeRetry 2m2s). Error: "MountVolume.SetUp failed for volume \"script\" (UniqueName: \"kubernetes.io/configmap/424b196e-5132-479a-8b95-63e41e0ea124-script\") pod \"create-pvc-3b132d90-8812-4391-bc29-966ee47bee0d\" (UID: \"424b196e-5132-479a-8b95-63e41e0ea124\") : configmap references non-existent config key: setup"
UPDATE - solution?
I changed the ConfigMap 'local-path-config' as described in the docs.
However, the jinja template in Kubespray lacks the properties 'setup' and 'teardown' in the configuration.
When I added de setup
and teardown
properties Kubernetes created the PV and the Pod started.
What's the reason Kubespray doesn't provide these properties in the template?