Kubernetes doesn't allow to mount file to container

2
votes

I ran into the below error when trying to deploy an application in a kubernetes cluster. It looks like kubernetes doesn't allow to mount a file to containers, do you know the possible reason?

deployment config file

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: model-loader-service
  namespace: "{{ .Values.nsPrefix }}-aai"
spec:
  selector:
    matchLabels:
      app: model-loader-service
  template:
    metadata:
      labels:
        app: model-loader-service
      name: model-loader-service
    spec:
      containers:
      - name: model-loader-service
        image: "{{ .Values.image.modelLoaderImage }}:{{ .Values.image.modelLoaderVersion }}"
        imagePullPolicy: {{ .Values.pullPolicy }}
        env:
        - name: CONFIG_HOME
          value: /opt/app/model-loader/config/
        volumeMounts:
        - mountPath: /etc/localtime
          name: localtime
          readOnly: true
        - mountPath: /opt/app/model-loader/config/
          name: aai-model-loader-config
        - mountPath: /var/log/onap
          name: aai-model-loader-logs
        - mountPath: /opt/app/model-loader/bundleconfig/etc/logback.xml
          name: aai-model-loader-log-conf
          subPath: logback.xml
        ports:
        - containerPort: 8080
        - containerPort: 8443
      - name: filebeat-onap-aai-model-loader
        image: {{ .Values.image.filebeat }}
        imagePullPolicy: {{ .Values.pullPolicy }}
        volumeMounts:
        - mountPath: /usr/share/filebeat/filebeat.yml
          name: filebeat-conf
        - mountPath: /var/log/onap
          name: aai-model-loader-logs
        - mountPath: /usr/share/filebeat/data
          name: aai-model-loader-filebeat
      volumes:
      - name: localtime
        hostPath:
          path: /etc/localtime
      - name: aai-model-loader-config
        hostPath:
          path: "/dockerdata-nfs/{{ .Values.nsPrefix }}/aai/model-loader/appconfig/"
      - name: filebeat-conf
        hostPath:
          path: /dockerdata-nfs/{{ .Values.nsPrefix }}/log/filebeat/logback/filebeat.yml

Details information of this issue:

message: 'invalid header field value "oci runtime error: container_linux.go:247:

        starting container process caused \"process_linux.go:359: container init

        caused \\\"rootfs_linux.go:53: mounting \\\\\\\"/dockerdata-nfs/onap/log/filebeat/logback/filebeat.yml\\\\\\\"

        to rootfs \\\\\\\"/var/lib/docker/aufs/mnt/7cd32a29938e9f70a727723f550474cb5b41c0966f45ad0c323360779f08cf5c\\\\\\\"

        at \\\\\\\"/var/lib/docker/aufs/mnt/7cd32a29938e9f70a727723f550474cb5b41c0966f45ad0c323360779f08cf5c/usr/share/filebeat/filebeat.yml\\\\\\\"

        caused \\\\\\\"not a directory\\\\\\\"\\\"\"\n"'

....

$ docker version
Client:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:38:45 2017
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.6
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   78d1802
 Built:        Tue Jan 10 20:38:45 2017
 OS/Arch:      linux/amd64

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.4", GitCommit:"793658f2d7ca7f064d2bdf606519f9fe1229c381", GitTreeState:"clean", BuildDate:"2017-08-17T08:48:23Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8+", GitVersion:"v1.8.3-rancher3", GitCommit:"772c4c54e1f4ae7fc6f63a8e1ecd9fe616268e16", GitTreeState:"clean", BuildDate:"2017-11-27T19:51:43Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
3
kubernetes

3 Answers

1
votes

caused "not a directory" is kind of self explanatory. What is the exact volume and volumeMount definition you use ? do you use subPath in your declaration ?

EDIT: change 

- name: filebeat-conf
  hostPath:
    path: /dockerdata-nfs/{{ .Values.nsPrefix }}/log/filebeat/logback/filebeat.yml

to 

- name: filebeat-conf
  hostPath:
    path: /dockerdata-nfs/{{ .Values.nsPrefix }}/log/filebeat/logback/

and add subPath: filebeat.yml to volumeMount

1
votes

SELinux may also be the culprit here. Log on to the node and execute sestatus. If the policy is disabled, you will see the output as SELINUX=disabled else it will be something similar to this:

SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             mcs
Current mode:                   permissive
Mode from config file:          permissive
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

First Option: You can either disable the selinux by editing the /etc/selinux/config file and update the SELINUX=permissive to SELINUX=disabled. Once done, reboot the machine and deploy to see if fixed. However, this is not the recommonded way and can be seen as a temporary fix.

Second Option: Log on to the node and execute ps -efZ | grep kubelet which will give something like this.

system_u:system_r:kernel_t:s0   root      1592     1  2 May23 ?        09:58:18 /usr/local/bin/kubelet --anonymous-auth=false

Now, from this output capture the string system_u:system_r:kernel_t:s0 which can be changed to security context as below in your deployment.

securityContext:
    seLinuxOptions:
      user: system_u
      role: system_r
      type: spc_t
      level: s0

Deploy your application and check the logs if it is fixed. Do let me know if this works for you or need any further help.

0
votes

Is this a multi-node cluster? If so, the file needs to exist on all Kubernetes nodes since the pod is typically scheduled on a randomly available host machine. In any case, ConfigMaps are a much better way to supply static/read-only files to a container.