Kubernetes w/ helm: MountVolume.SetUp failed for volume "secret" : invalid character '\r' in string literal

Question

I'm using a script to run helm command which upgrades my k8s deployment.
Before I've used kubectl to directly deploy, as I've move to helm and started using charts, I see an error after deploying on the k8s pods:

MountVolume.SetUp failed for volume "secret" : invalid character '\r' in string literal

My script looks similar to:

value1="foo"
value2="bar"
helm upgrade deploymentName --debug --install --atomic --recreate-pods --reset-values --force --timeout 900 pathToChartDir --set value1 --set value2

The deployment.yaml is as following:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: deploymentName
spec:
  selector:
    matchLabels:
      run: deploymentName
  replicas: 2
  template:
    metadata:
      labels:
        run: deploymentName
        app: appName
    spec:
      containers:
      - name: deploymentName
        image: {{ .Values.image.acr.registry }}/{{ .Values.image.name }}:{{ .Values.image.tag }}
        volumeMounts:
        - name: secret
          mountPath: /secrets
          readOnly: true
        ports:
        - containerPort: 1234
        env:
        - name: DOTENV_CONFIG_PATH
          value: "/secrets/env"
      volumes:
      - name: secret
        flexVolume:
          driver: "azure/kv"
          secretRef:
            name: "kvcreds"
          options:
            usepodidentity: "false"
            tenantid: {{ .Values.tenantid }}
            subscriptionid: {{ .Values.subsid }}
            resourcegroup: {{ .Values.rg }}
            keyvaultname: {{ .Values.kvname }}
            keyvaultobjecttype: secret
            keyvaultobjectname: {{ .Values.objectname }}

As can be seen, the error relates to the secret volume and its values.
I've triple checked there is no line-break or anything like that in the values.

I've run helm lint - no errors found.

I've run helm template - nothing strange or missing in output.

Update:
I've copied the output of helm template and put in a deploy.yaml file.
Then used kubectl apply -f deploy.yaml to manually deploy the service, and... it works.
That makes me think it's actually some kind of a bug in helm? make sense?

Update 2:
I've also tried replacing the azure/kv volume with emptyDir volume and I was able to deploy using helm. It looks like a specific issue of helm with azure/kv volume?

Any ideas for a workaround?

Greg Oliver Greg Oliver · Accepted Answer · 2019-08-22T08:37:59

A completely correct answer requires that I say the actual details of your \r problem might be different from mine.

I found the issue in my case by looking in the kv log of the AKS node (/var/log/kv-driver.log). In my case, the error was:

Original Error: autorest/azure: Service returned an error. Status=403 Code="Forbidden" Message="Access denied. Caller was not found on any access policy.\r\n

You can learn to SSH into the node on this page:

https://docs.microsoft.com/en-us/azure/aks/ssh

If you want to follow the solution, I opened an issue:

https://github.com/Azure/kubernetes-keyvault-flexvol/issues/121

Kubernetes w/ helm: MountVolume.SetUp failed for volume "secret" : invalid character '\r' in string literal

1 Answers