Get IP addresses of all k8s pods from within a container

Question

I have a service called my-service with an endpoint called refreshCache. my-service is hosted on multiple servers, and occasionally I want an event in my-service on one of the servers to trigger refreshCache on my-service on all servers. To do this I manually maintain a list of all the servers that host my-service, pull that list, and send a REST request to <server>/.../refreshCache for each server.

I'm now migrating my service to k8s. Similarly to before, where I was running refreshCache on all servers that hosted my-service, I now want to be able to run refreshCache on all the pods that host my-service. Unfortunately I cannot manually maintain a list of pod IPs, as my understanding is that IPs are ephemeral in k8s, so I need to be able to dynamically get the IPs of all pods in a node, from within a container in one of those pods. Is this possible?

Note: I'm aware this information is available with kubectl get endpoints ..., however kubectl will not be available within my container.

Answer 1

For achieving this the best way would be to use a K8s config inside the pod.

For this the K8s Client can help. Here is an example python script that can be used to get pods and their metadata from inside the pod.

from kubernetes import client, config


def trigger_refresh_cache():
    # it works only if this script is run by K8s as a POD
    config.load_incluster_config()

    v1 = client.CoreV1Api()
    print("Listing pods with their IPs:")
    ret = v1.list_pod_for_all_namespaces(label_selector='app=my-service')
    for i in ret.items:
        print("%s\t%s\t%s" %
              (i.status.pod_ip, i.metadata.namespace, i.metadata.name))
        # Rest of the logic goes here to trigger endpoint

Here the method load_incluster_config() is used which loads the kubeconfig inside pod via the service account attached to that pod.

Answer 2

You don't need kubectl to access the Kubernetes API. You can do it with any tool that can make HTTP requests.

The Kubernetes API is a simple HTTP REST API, and all the authentication information that you need is present in the container if it runs as a Pod in the cluster.

To get the Endpoints object named my-service from within a container in the cluster, you can do:

curl -k -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  https://kubernetes.default.svc:443/api/v1/namespaces/{namespace}/endpoints/my-service

Note: replace {namespace} with the namespace of the my-service Endpoints resource.

And to extract the IP addresses of the returned JSON, you could pipe the output to a tool like jq:

... | jq -r '.subsets[].addresses[].ip'

Note that the Pod from which you are executing this needs read permissions for the Endpoints resource, otherwise the API request is denied.

You can do this with a ClusterRole, ClusterRoleBinding, and Service Account (you need to set this up only once):

kubectl create sa endpoint-reader
kubectl create clusterrole endpoint-reader --verb=get,list --resource=endpoints
kubectl create clusterrolebinding endpoint-reader --serviceaccount=default:endpoint-reader --clusterrole=endpoint-reader

Then, use the endpoint-reader ServiceAccount for the Pod from which you want to execute the above curl command by specifying it in the pod.spec.serviceAccountName field.

Granting permissions for any other API operations (i.e. combinations of verbs and resources) works in the same way.