How to find Kubernetes pod that's handling the request

Question

I am running 2 pods and a service with Type: NodePort, to load balance the requests between pods. I want to know when I send a request to the service, which pod the request is forwarded to. Is there a way to find this, because looking at the response, it looks like all the requests are handled by same pod.

Answer 1

A kubernetes Service will load-balance using WRR by default. When you create a Service, iptables rules will be generated in the node.

To be sure, ssh into the node and run iptables-save|less. Search for the name of the service. In the example below, a Service microbot load balances the microbot deployment with 3 replicas. There should be 2 entries in your case since you have just 2 pods.

-A KUBE-SVC-LX5ZXALLN4UQ7ZFL -m comment --comment "default/microbot:" -m statistic --mode random --probability 0.33332999982 -j KUBE-SEP-OZCDYTQTC3KQGJK5
-A KUBE-SVC-LX5ZXALLN4UQ7ZFL -m comment --comment "default/microbot:" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-SKIRAXBCCQB5R4MV
-A KUBE-SVC-LX5ZXALLN4UQ7ZFL -m comment --comment "default/microbot:" -j KUBE-SEP-SPMPNZCOIJIRSNNQ

If the iptables output doesn't look like the above, it's likely that your Service is not configured properly. Like what Heidi said, the pods are not associated with the Service.

Answer 2

1. You can look at the application log file, if your using one. if you print anything in stdout, use kubectl logs <pod> to see the message.

2. For testing you can include pod hostname in the response.

Answer 3

Most likely because of your context and lack of namespace you can't see the logs.

Try kubectl get pods -o wide --all-namespaces | grep <pod> to get the information of which namespace the pod is in and the node IP address.

Then feed the and into the command below to get a running tail of the last 100 lines of the log kubectl --namespace <namespace> logs --tail 100 -f <pod>

There is also the remote chance that the pod is not associated with the service. To check run kubectl describe services --namespace <namespace> <service> and look for the app name in the Selector: section

You can also exec into the container and see if the port is accessible or bound on the pod itself. If it isn't listening or answering, most likely it's due to the service not being associated with the application in the namespace.