How would I run a simple container the triggers Prometheus to reload its config when the underlying ConfigMap changes?

1
votes

While running Prometheus in Kubernetes, I'm pushing out new config via a ConfigMap. ConfigMaps are exposed as files in the container.

I would love Prometheus to automatically reload its configuration when the file changes.

Would something like this work?

inotifywait -q -m -e close_write /etc/prometheus/config.yml |
while read -r filename event; do
  curl -X POST http://localhost:9090/-/reload
done
2
kubernetesprometheus
I've updated my answer - I actually found it quite interesting to have a fully working solution which can be used by everyone. Would be cool to hear if that works for you too – pagid

2 Answers

2
votes

(Edit: I took some time to get this fully to work) this works with a small sidecar container. The configuration could look like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus
spec:
  replicas: 1
  template:
    ....
    spec:
      containers:
        ... (your actual container config goes here) ...
      - name: refresh
        imagePullPolicy: Always
        args:
        - /etc/prometheus/config.yml
        - http://localhost:9090/-/reload
        image: tolleiv/k8s-prometheus-reload
        volumeMounts:
          - name: config-volume
            mountPath: /etc/prometheus
      volumes:
        - name: config-volume
          configMap:
            name: prometheus

The actual check is done with this script where the observed file and the URL are passed as parameters:

#!/bin/sh
while true; do
   inotifywait "$(readlink -f $1)"
   echo "[$(date +%s)] Trigger refresh"
   curl -sSL -X POST "$2" > /dev/null
done

Everything can be found in this container on Dockerhub

Keeping a single inotifywait with the -m didn't work because of symlink juggling which is done by Kubernetes when the ConfigMap changes.

0
votes

Another option would be to use a livenessProbe command and just trigger a restart of the Pod when the configuration changed.

That could look like this:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
      - name: prometheus
        image: prom/prometheus:latest
        ports:
        - containerPort: 9090
        volumeMounts:
          - name: config-volume
            mountPath: /etc/prometheus
        livenessProbe:
          exec:
            command:
            - /bin/sh
            - -c
            - "test -z $(find /etc/prometheus -mmin -2)"
          initialDelaySeconds: 300
          periodSeconds: 10
      volumes:
        - name: config-volume
          configMap:
            name: prometheus

A drawback could be that this way you'd loose the data cached in memory, but it's straight forward and doesn't need a sidecar container.