endpoint is exposes but trying to use netstat from another pod in the same namespace fails to reach it

Question

Any help will be appreciated.

I'm trying to send a request from pod to pod inside a k8s cluster in the same namespace running on AWS.

I've added a pod that serves requests coming from port 5000, from within the pod I manage to run a curl command successfully:

curl -i \
  --header "Content-Type: application/json" \
  --request POST \
  --data '[[5.1, 3.5, 1.4, 0.2]]' \
  localhost:5000/predict

I've added a service to the pod to expose an endpoint so I can send traffic to it:

apiVersion: v1
kind: Service
metadata:
  annotations:
  name: sklearn-iris-bentoml-deployment-sklearn-iris-example
  namespace: bentoml
  ownerReferences:
  selfLink: /api/v1/namespaces/bentoml/services/sklearn-iris-bentoml-deployment-sklearn-iris-example
  labels:
    run: my-bentoml
spec:
  ports:
  - name: http
    port: 5000
    protocol: TCP
    targetPort: 5000
  selector:
    run: my-bentoml
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

The service:

➜ kubectl get svc -n bentoml
NAME                                                   TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
sklearn-iris-bentoml-deployment-sklearn-iris-example   ClusterIP   10.110.5.245   <none>        5000/TCP   5h6m

The endpoint:

➜ k get endpoints sklearn-iris-bentoml-deployment-sklearn-iris-example -n bentoml
NAME                                                   ENDPOINTS             AGE
sklearn-iris-bentoml-deployment-sklearn-iris-example   10.110.115.108:5000   5h6m

When I try to connect through telnet from a different pod in the same namespace, it fails:

root@my-shell:/# telnet 10.110.115.108 5000
Trying 10.110.115.108...
telnet: Unable to connect to remote host: Connection refused

I'm not sure whether I should configure a DNS since I don't have intentions to send traffic from outside the cluster so the network is mainly focused inside the cluster

Answer 1

Make sure the the application pod is listening on 0.0.0.0 instead of 127.0.0.1 or localhost. This can typically be changed in the code of the application and depends on the language of the application.

Answer 2

This is the netstat -an output: tcp 0 0 127.0.0.1:5000 0.0.0.0:* LISTEN – torpido 3 hours ago

It confirms the correctness of @Arghya Sadhu suspicions that your issue is related with the fact that your application is configured to listen only on localhost address which means it can be reached only locally. It would be seen more clearly if you pasted your netstat output with the headlines describing the content of each column.

Just take a look at an example. To illustrate the idea I deployed a set of nginx Pods from the example available in the official k8s docs.

After listing the available pods and attaching to one of them via kubectl exec I ran netstat -ntlp command:

root@nginx-deployment-pod1:/# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1/nginx: master pro

As you can see, the output in the third column is quite different than in your example. 0 0.0.0.0:80 means that the nginx server listens on all IPv4 addresses, assigned to all network interfaces in this system (in this case container or Pod). You can list all the available network interfaces and addresses with ip a or ifconfig commands.

Take a look at your command:

curl -i \
  --header "Content-Type: application/json" \
  --request POST \
  --data '[[5.1, 3.5, 1.4, 0.2]]' \
  localhost:5000/predict

You specified that you want to reach your app on port 5000 on localhost, but the fact it is available on localhost doesn't mean it will be reachable also outside the Pod. Let's take a look again at your netstat output (this time with headers):

Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp   0      0      127.0.0.1:5000          0.0.0.0:*               LISTEN

As you can see, your app listens or in other words binds only to localhost address (127.0.0.1). As @Arghya Sadhu mentioned, it is typically configured somewhere in the application code or in the configuration files if such are provided.

Keep in mind that you can expose some port further via Services and Endpoints but they must be first exposed by the Pod, which means the application which is running withing the Pod must be configured to listen on all local IP addresses.

Going back to our nginx example, it listens on 0.0.0.0:80 because it is configured to do so. It is done by the following directive present in /etc/nginx/conf.d/default.conf configuration file:

server {
    listen       80;
    server_name  localhost;
...

or more specifically this line:

listen       80;

It equels to:

listen       0.0.0.0:80;

It basically means "listen on all IP addresses / all network interfaces on port 80". If we changed it to:

listen       localhost:80;

or

listen       127.0.0.1:80;

nginx will be listening only on loclhost (127.0.0.1) address and it will be reachable only locally i.e. from within the Pod.

Every application does this in it's own way. If you deploy on kubernetes some custom application, which doesn't come with configuration files that enable you to defind listen or bind IP address, chances are it is hardcoded in the application code, so you need to find the fragment which is responsible for it. Look specifically for entries containing either localhost or 127.0.0.1 and change it to 0.0.0.0 which means all IP addresses, searching for lines containing listen or bind keywords may be also helpful.