Service IP is not accessible across nodes in kubernetes

Question

I have created a kubernetes v1.2 running in Azure cloud with one master(Master) and two nodes(Node1 and Node2). I have deployed an Nginx and Tomcat application. Both the containers are deployed in individual pods with RC and they have a SERVICE for each.

Nginx pod is deployed in the Node1 and Tomcat pod is deployed in Node2. Now Nginx from Node1 is trying to access Tomcat via tomcat's ServiceIP(clusterIP) which is in Node2. But its unreachable.

Nginx serviceIP: 10.16.0.2 Node1

Tomcat serviceIP: 10.16.0.4 Node2

I tried curl 10.16.0.4:8080 from Node2, it works. But same from Node1 fails with curl: (52) Empty reply from server

So communication to serviceIP across nodes fails. Is this the problem with kube v1.2?

Note: ClusterIP for the Service will be specified at the time of creating the service.

Can you reach the Tomcat pod's IP directly? That will help determine whether it is a service routing problem or a more general networking issue. — CJ Cullen
Which overlay network plugin are you using ? . Are you using iptables mode or userspace mode ? Please refer this link stackoverflow.com/questions/36088224/… — atv
@CJCullen I can reach the pods via its IP from the same node and not from the different node. Curl 10.32.0.2:8080 from Node2 reaches Tomcat, same from Node1 fails — Sujai Sivasamy
@MrE I don't have DNS setup(no addons installes). Do we need DNS setup for inter-node communication? — Sujai Sivasamy

Sreekanth Pothanis Sreekanth Pothanis · Accepted Answer · 2016-08-30T22:02:58

Since you are able to reach the cluster ip from the Node2, it looks like the service selector is properly defined.

Kube-proxy is the component that watches the services and creates iptable rules for end points. I would check if kube-proxy is running properly on Node1. Then check if iptable rules are set properly for the cluster ip you are trying to reach. You can see these with iptables -L -t nat | grep namespace/servicename

Here is an example:

bash-4.3# iptables -L -t nat | grep kube-system/heapster KUBE-MARK-MASQ all -- 172.168.16.182 anywhere /* kube-system/heapster: */ DNAT tcp -- anywhere anywhere /* kube-system/heapster: */ tcp to:172.168.16.182:8082 KUBE-SVC-BJM46V3U5RZHCFRZ tcp -- anywhere 192.168.172.66 /* kube-system/heapster: cluster IP */ tcp dpt:http KUBE-SEP-KNJP5BBKUOCH7NDB all -- anywhere anywhere /* kube-system/heapster: */

In this example I looked up heapster running in kube-system namespace. It is showing that the cluster ip is 192.168.172.66 DNATs to the endpoint 172.168.16.182, which is pods ip (You should cross check this with the endpoints listed in kubectl describe service.

If is it not there, restarting kube-proxy might help.

Service IP is not accessible across nodes in kubernetes

1 Answers