Refresh daemonset pods when configmap updates in Kubernetes

Question

How do I automatically restart Kubernetes pods associated with Daemonsets when their configmap is updated?

As per the kubernetes documentation when the configmap volume mount is updated, it automatically updates the pods. However I do not see that happening for Daemonsets. What am I missing?

The below is my configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-update
  namespace: default
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf


    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-logaggregator.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/abc.log
        Parser            docker
        DB                /var/log/tail-containers-state.db
        DB.Sync           Normal
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
        Rotate_Wait      60
        Docker_Mode      On

  filter-kubernetes.conf: |

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.conatiners.
        Merge_Log           On
        Keep_Log            Off
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Labels              On
        Annotations         On

  output-kubernetes.conf: |

    [OUTPUT]
        Name              cloudwatch
        Match             kube.*
        region            us-west-2
        log_group_name    fluent-bit-cloudwatch
        log_stream_prefix from-fluent-bit
        auto_create_group true

  parsers.conf: |

    [PARSER]
        Name         docker
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On
        Decode_Field_As   escaped_utf8    log    do_next
        Decode_Field_As   json       log

    [PARSER]
        Name         docker_default
        Format       json
        Time_Key     time
        Time_Format  %Y-%m-%dT%H:%M:%S.%L
        Time_Keep    On

& my daemonset manifest file

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluent-bit-update
  namespace: default
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
    spec:
      containers:
        - name: aws-for-fluent-bit
          image: amazon/aws-for-fluent-bit:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 2020
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: fluent-bit-volume
              mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: fluent-bit-volume
          configMap:
            name: fluent-bit-update
      serviceAccountName: fluent-bit
      tolerations:
        - key: node-role.kubernetes.io/master
          operator: Exists
          effect: NoSchedule
        - operator: "Exists"
          effect: "NoExecute"
        - operator: "Exists"
          effect: "NoSchedule"

When I updated the Path field in configmap to read another log file, though I see the volume mounts getting updated, I do not see the pods picking up the change unless I delete and recreate the daemonset.

Is there a way to achieve this automagically without restarting the daemonset? I would appreciate some guidance on this. Thanks

What do you mean about "I do not see the pods picking up the change". Kubernetes is only responsible to update the volume with the updated config map. — alidadar7676

anmol agrawal anmol agrawal · Accepted Answer · 2020-06-11T07:23:16

As per the documentation you pointed, Kubernetes will update the configmap (fluent-bit configuration in your case) but it is your application's duty to pick the updated config. Generally, the application takes the config during start-time but to update the config periodically, either your application should support this or another module (like a config updater) should be used which will restart your app (without restarting pod) when there is a change in config. For fluent-bit you can refer this github issue for dynamic configuration. Prometheus already supports it like this.

Refresh daemonset pods when configmap updates in Kubernetes

1 Answers