Creating Prometheus metrics from logs of containers which are running in Kubernetes on Google Cloud

Question

I am new to prometheus and am wondering how I should approach the following problem.

I need to monitor some applications that are running as containers in Kubernetes. I need to create custom metrics using the logs of my containers (which are the applications logs) in Kubernetes on GKE, using Prometheus. For example I need to count how many logs of a specific container contain a particular word (using regex). Logs are output to stdout of each container and also are collected on Stackdriver in GKE.

What is the best way of creating my metrics out of the container logs, to monitor my container using Prometheus? Do I need an exporter or should I directly instrument my containers to expose the metrics on a http endpoint to Prometheus? or should I instrument my Stackdriver and expose the metrics there to Prometheus.

I am trying to make sure which one of the above approaches are possible and make sense. I would appreciate any advice.

Answer 1

Actually all approaches should be possible, but I'd definitely recommend instrumenting your application directly as it involves the least overhead are sources of error.

If instrumenting your application is not possible you can create a custom StackDriver metrics, export that to Prometheus and alert on that. But I would definitely recommend instrumenting your applications directly.