Kubernetes - How to dynamically refresh secrets without restarting pod

Question

I have a kubernetes cluster with vault installed (by a helm chart).

I want to populate secret from vault to a file in a pod (nginx for example) and refresh the secrets every 5 minutes.

I used the following configuration to test it (with appropriate vault policy/backend auth):

namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: web

Service_account.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx
  namespace: web
secrets:
- name: nginx

nginx-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: web
  labels:
    app: nginx
    run: nginx
    version: vault-injector
spec:
  replicas: 1
  selector:
    matchLabels:
      run: nginx
      version: vault-injector
  template:
    metadata:
      labels:
        app: nginx
        run: nginx
        version: vault-injector
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "nginx"
        #vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/agent-inject-secret-nginx.pass: "infrastructure/nginx/"
    spec:
      serviceAccountName: nginx
      containers:
        - name: nginx
          image: nginx
          ports:
            - name: http
              containerPort: 80

When i apply this configuration to my kubernetes cluster the deployment is created and my secret are filled into /vault/secret/nginx.pass(as expected).

kubectl exec -it pod/nginx-69955d8744-v9jm2 -n web -- cat /vault/secrets/nginx.pass
Password1: MySecretPassword1
Password2: MySecretPassword2

I tried to update the kv and add a password on nginx kv but my pods doesn't refresh the file on /vault/secrets/nginx.pass. If i restart my secrets are filled

Is it possible to dynamically refresh the kv ? What's the best way to do it ? I want to use vault as a configuration manager and be able to modify kv without restarting pods.

Maybe this can help you stackoverflow.com/questions/37317003/… — paltaa

tzouintzouin tzouintzouin · Accepted Answer · 2020-09-06T07:51:18

You can define a TTL on your kv secret by specifying a TTL value. For example :

 vault kv put infrastructure/nginx ttl=1m Password1=PasswordUpdated1 Password2=PasswordUpdated2

will expire your infrastructure/nginx secret every minute. Vault sidecar will automatically check for a new value and refresh the file into your pods.

root@LAP-INFO-28:/mnt/c/Users/cmonsieux/Desktop/IAC/kubernetes/yaml/simplePod# k logs nginx-69955d8744-mwhmf vault-agent -n web
    renewal process
    2020-09-06T07:16:42.867Z [INFO]  sink.file: token written: path=/home/vault/.vault-token
    2020-09-06T07:16:42.867Z [INFO]  template.server: template server received new token
    2020/09/06 07:16:42.867793 [INFO] (runner) stopping
    2020/09/06 07:16:42.867869 [INFO] (runner) creating new runner (dry: false, once: false)
    2020/09/06 07:16:42.868051 [INFO] (runner) creating watcher
    2020/09/06 07:16:42.868101 [INFO] (runner) starting
    2020-09-06T07:16:42.900Z [INFO]  auth.handler: renewed auth token
    2020/09/06 07:18:26.268835 [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/nginx.pass"
    2020/09/06 07:19:18.810479 [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/nginx.pass"
    2020/09/06 07:24:41.189868 [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/nginx.pass"
    2020/09/06 07:25:36.095547 [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/nginx.pass"
    2020/09/06 07:29:11.479051 [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/nginx.pass"
    2020/09/06 07:31:00.715215 [INFO] (runner) rendered "(dynamic)" => "/vault/secrets/nginx.pass"
root@LAP-INFO-28:/mnt/c/Users/cmonsieux/Desktop/IAC/kubernetes/yaml/simplePod# k exec -it pod/nginx-69955d8744-mwhmf -n web -- cat /vault/secrets/nginx.pass
Password1: PasswordUpdated1
Password2: PasswordUpdated2
ttl: 1m

Kubernetes - How to dynamically refresh secrets without restarting pod

1 Answers