CephFS Volume Mount Fails

Question

I've successfully deployed a Kubernetes cluster using the docker-multinode configuration as well as a Ceph cluster and am able to mount a CephFS device manually using the following:

sudo mount -t ceph monitor1:6789:/ /ceph -o name=admin,secretfile=/etc/ceph/cephfs.secret

I'm now attempting to launch a pod using the kubernetes example here:

apiVersion: v1
kind: Secret
metadata:
  name: ceph-secret
data:
  key: my-ceph-secret-key
---
apiVersion: v1
kind: Pod
metadata:
  name: cephfs2
spec:
  containers:
  - name: cephfs-rw
    image: kubernetes/pause
    volumeMounts:
    - mountPath: "/mnt/cephfs"
      name: cephfs
  volumes:
  - name: cephfs
    cephfs:
      monitors:
      - "monitor1:6789"
      - "monitor2:6789"
      - "monitor3:6789"
      user: admin
      secretRef:
        name: ceph-secret
      readOnly: false

When I run:

sudo kubectl create -f cephfs.yml

I am receiving the following error:

Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/cephfs/445ee063-d1f1-11e6-a3e3-1418776a29a6-cephfs" (spec.Name: "cephfs") pod "445ee063-d1f1-11e6-a3e3-1418776a29a6" (UID: "445ee063-d1f1-11e6-a3e3-1418776a29a6") with: CephFS: mount failed: mount failed: fork/exec /bin/mount: invalid argument Mounting arguments: monitor1:6789,monitor2:6789,monitor3:6789:/data /var/lib/kubelet/pods/445ee063-d1f1-11e6-a3e3-1418776a29a6/volumes/kubernetes.io~cephfs/cephfs ceph [name=admin,secret=secret]

Do the kubernetes manager containers need to have the ceph-fs-common package installed in order to perform a successful mount? I cannot find any further debugging information to determine the cause of the error.

Answer 1

AFAIK you might have 2 problems here:

• Ceph required the ip addresses of the machines to work
• The OS you are running the container on, is the one which mounts the storage: The ceph tooling needs to be installed on that machine. The container is completely unaware of the mounted disks

Answer 2

There were a couple of issues that needed fixed in order to successfully mount a CephFS volume in kubernetes. Keep in mind I've deployed Kubernetes 1.4.6 using the kube-deploy docker multinode configuration.

Issue #1: Mount command Fails using Kubernetes secrets

When examining the error above more closely, I found that Kubernetes encrypts my Ceph secret with characters that are interpreted as newlines. As a result, the kubelet fails when attempting to mount the file system.

To workaround, I configured my YAML to use a Ceph secretfile instead of a Kubernetes secret:

apiVersion: v1
kind: Pod
metadata:
  name: cephfs-test
spec:
  containers:
  - name: cephfs-rw
    image: kubernetes/pause
    volumeMounts:
    - mountPath: "/mnt/cephfs"
      name: cephfs
  volumes:
  - name: cephfs
    cephfs:
      monitors:
      - "<monitor1>:6789"
      - "<monitor2>:6789"
      - "<<monitor3>:6789"
      user: admin
      # Omit for CephFS mounting error
      # secretRef:
      #   name: ceph-secret
      secretFile: "/etc/ceph/user.secret"
      readOnly: false

Issue #2: Kubelet Missing Ceph Packages and Configuration

The kubelets were all missing the ceph-fs-common and ceph-common packages required to mount CephFS volumes to containers as well as the necessary configuration files. The following script should applies the necessary updates to the kubelet master/worker agents:

docker exec $KUBELET_ID apt-get update
docker exec $KUBELET_ID apt-get install -y wget lsb-release apt-transport-https
docker exec $KUBELET_ID /bin/bash -c "wget -q -O- 'https://download.ceph.com/keys/release.asc' | apt-key add -"
RELEASE=$(docker exec $KUBELET_ID lsb_release -sc)
docker exec $KUBELET_ID /bin/bash -c "echo deb https://download.ceph.com/debian-jewel/ $RELEASE main | tee /etc/apt/sources.list.d/ceph.list"
docker exec $KUBELET_ID apt-get update
docker exec $KUBELET_ID apt-get install -y ceph-fs-common ceph-common

docker exec $KUBELET_ID mkdir -p /etc/ceph
docker exec $KUBELET_ID /bin/bash -c "echo $CEPH_SECRET > /etc/ceph/admin.secret"

Full gist here