Pod cannot connect to itself via service IP (kubernetes v1.10)

Question

I have a kubernetes cluster (v1.10) and flannel as cni. Using the default settings, the setup works fine (using kubeadm) but a pod cannot connect to itself via service IP.

Tried setting hairpin-mode as "promiscuous-bridge" but kubelet complains:

Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"

Not sure what's going on. Went through this https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/#a-pod-cannot-reach-itself-via-service-ip but no help. Any pointers ?

It's usually something wrong with iptables. First check whether kube-proxy is up. Run sudo docker ps | grep kube-proxy on the node where the pod resides. — ccshih
@ffledgling Pod can communicate with other pods and services. The only problem is connecting to itself via it's service IP. — Phagun Baya
@PhagunBaya The question was whether other pods can communicate with this pod via it's Service IP. If not, networking on this pod/node is screwed up. If they can, it's probably a missing route addition by your networking plugin/custom config on the base machines. — ffledgling

Phagun Baya Phagun Baya · Accepted Answer · 2018-04-28T20:02:27

The default flannel configuration does not set HairpinMode to true.

https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

Setting hairpinMode to true in flannel config resolved the issue. The config change is as below:

kind: ConfigMap
apiVersion: v1
metadata:
  name: kube-flannel-cfg
  namespace: kube-system
  labels:
    tier: node
    app: flannel
data:
  cni-conf.json: |
    {
      "name": "cbr0",
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    }
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

Complete flannel config is here:

https://gist.githubusercontent.com/phagunbaya/2a53519a9427ba0623244f1680a5b5ff/raw/13ada0d6dd92388c8c5aae93bfb1ccaf9c79f60b/flannel-0.9.1.yaml

Instead of the default kubectl command to apply flannel cni use following command:

kubectl apply -f https://gist.githubusercontent.com/phagunbaya/2a53519a9427ba0623244f1680a5b5ff/raw/13ada0d6dd92388c8c5aae93bfb1ccaf9c79f60b/flannel-0.9.1.yaml

Pod cannot connect to itself via service IP (kubernetes v1.10)

2 Answers