Get data from prometheus only from last scrape iteration

Question

Use case

I am creating a tool to show the resource requests of all pods in a kubernetes cluster. Prometheus already scrapes my prometheus metric exporter and historically saves all the scraped metrics (interval 10s), so that I could even see the resource requests from all pods 10 days ago.

The problem

Every time a pod has been redeployed it does change the name. However I only want to show the resource requests of actually RUNNING pods. For this purpose I could obviously add a time selector like, show me only max 15s old results, but that would be a little hack. If possible I'd like to get the results from the latest scrape iteration (which in case of problems could be older than 15s).

The question

What's the best way to get only those prometheus metrics which have been scraped in the last iteration?

eagle_container_resource_requests_cpu for instance would return the requested CPU cores of pods which nowadays no longer exist.

Side note: I am trying to achieve this behaviour in a Grafana table.

Answer 1

Staleness might be what you are looking for: https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness

So after 5 minutes you should begin to see values disappearing.

Answer 2

When you query your metric eagle_container_resource_requests_cpu you get the latest value, e.g. this is an actual request you send by using Prometheus graph execution:

https://<prometheus-url>/api/v1/query?query=eagle_container_resource_requests_cpu&time=1541769935.01&_=1541769677378

More over, you can verify it with directly calling Prometheus API:

curl 'http://localhost:9090/api/v1/query?query=eagle_container_resource_requests_cpu&time=2018-11-09T10:10:51.781Z'

Both responses will contain the time stamp metric was scraped, e.g.:

"value":[1541770056.042,"11"]

If you would like to see this in Grafana's singlestat chart:

You can either see the "current" (the latest) value, or time of this last point.