How to create a ServiceMonitor for prometheus-operator?

Question

Recently, prometheus-operator has been promoted to stable helm chart (https://github.com/helm/charts/tree/master/stable/prometheus-operator).

I'd like to understand how to add a custom application to monitoring by prometheus-operator in a k8s cluster. An example for say gitlab runner which by default provides metrics on 9252 would be appreciated (https://docs.gitlab.com/runner/monitoring/#configuration-of-the-metrics-http-server).

I have a rudimentary yaml that obviously doesn't work but also not provides any feedback on what isn't working:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: gitlab-monitor
  # Change this to the namespace the Prometheus instance is running in
  namespace: default
  labels:
    app: gitlab-runner-gitlab-runner
    release: prometheus
spec:
  selector:
    matchLabels:
      app: gitlab-runner-gitlab-runner
  namespaceSelector:
    # matchNames:
    # - default
    any: true
  endpoints:
  - port: http-metrics
    interval: 15s

This is the prometheus configuration:

> kubectl get prometheus -o yaml

...
serviceMonitorNamespaceSelector: {}
serviceMonitorSelector:
  matchLabels:
    release: prometheus
...

So the selectors should match. By "not working" I mean that the endpoints do not appear in the prometheus UI.

Could you please post the output of kubectl get prometheus -o yaml. There you have to specify serviceMonitorNamespaceSelector: {} and serviceMonitorSelector: {} (example will select all serviceMonitors in all namespaces). Your ServiceMonitor definition looks fine to me. — Peter Ittner
could you please define serviceMonitorSelector: {} just to exclude that this selector does not work properly. You can also check the annotations of your services. I have a selector matchLabels: <key>: <value> in my case and it does work. I'm not sure if namespaceSelector: # matchNames: # - default any: true also works. — Peter Ittner
Based on the monitors shipped with prometheus operator I can even drop the namespaceSelector altogether (though I've tried both approaches). Not showing up in prometheus targets. Is there any logfile I could check for where service monitors are getting picked up? — andig
Bang head against wall. Looking at github.com/coreos/prometheus-operator/blob/… I need a service too, not only a service monitor? — andig

andig andig · Accepted Answer · 2018-10-26T16:36:27

Thanks to Peter who showed me that it idea in principle wasn't entirely incorrect I've found the missing link. As a servicemonitor does monitor services (haha), I missed the part of creating a service which isn't part of the gitlab helm chart. Finally this yaml did the trick for me and the metrics appear in Prometheus:

# Service targeting gitlab instances
apiVersion: v1
kind: Service
metadata:
  name: gitlab-metrics
  labels:
    app: gitlab-runner-gitlab-runner
spec:
  ports:
  - name: metrics # expose metrics port
    port: 9252 # defined in gitlab chart
    targetPort: metrics
    protocol: TCP
  selector:
    app: gitlab-runner-gitlab-runner # target gitlab pods
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: gitlab-metrics-servicemonitor
  # Change this to the namespace the Prometheus instance is running in
  # namespace: default
  labels:
    app: gitlab-runner-gitlab-runner
    release: prometheus
spec:
  selector:
    matchLabels:
      app: gitlab-runner-gitlab-runner # target gitlab service
  endpoints:
  - port: metrics
    interval: 15s

Nice to know: the metrics targetPort is defined in the gitlab runner chart.

How to create a ServiceMonitor for prometheus-operator?

2 Answers