Find Pod Count in splunk search

Question

I have written below query to fetch the pod count for a particular application as below:

index="stream_data" sourcetype="kube:container:stream_app" | search pod="data-app-streams-*" | stats dc(pod) as pod_count

I have set the duration for this alert as 15 min. However, the pod count shown by this query comes to 2 because it gives the logs for the busy pods only. But the actually running pods are 4. Here pod is the already available field.

Can anyone suggest another approach for finding the actual pod count

There's not a lot to work with in this question. What's the difference between a "busy" pod and a "running" pod? — RichG
@RichG basically not all the pods emit the logs...all the time... that is why splunk is not able to detect the correct number of pods running..is there any way to figure this out?? — knowledge20

RichG RichG · Accepted Answer · 2021-04-11T23:54:05

Splunk can only work with the data it has. If there are 4 pods, but only 2 of them have sent an event recently then Splunk will think there are only 2 pods. One answer is search back further than 15 minutes. Even then you run the risk of not all pods reporting an event during the time searched. There's not much one can do about that.

Find Pod Count in splunk search

1 Answers