Kubernetes: Unable to access flannel pod

Question

I am new to Kubernetes. I have set up 3 Ubuntu 20.04.2 LTS VMs on Oracle Virtualbox Manager.

I have installed docker, kubelet, kubeadm, and kubectl in all 3 VMs according to the following documentation.
https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/

And I created cluster using the following link: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/

I used the following commands to setup flannel

$ wget https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml
$ kubectl create -f kube-flannel.yml

Everything looks fine.

root@master-node:~/k8s# kubectl get nodes -o wide
NAME          STATUS   ROLES                  AGE   VERSION   INTERNAL-IP      EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
master-node   Ready    control-plane,master   23h   v1.20.5   192.168.108.10   <none>        Ubuntu 20.04.2 LTS   5.4.0-70-generic   docker://19.3.15
node-1        Ready    <none>                 10h   v1.20.5   192.168.108.11   <none>        Ubuntu 20.04.2 LTS   5.4.0-70-generic   docker://19.3.15
node-2        Ready    <none>                 10h   v1.20.5   192.168.108.12   <none>        Ubuntu 20.04.2 LTS   5.4.0-70-generic   docker://19.3.15

I then create nginx deployment with 3 replicas.

root@master-node:~/k8s# kubectl get po -o wide
NAME                            READY   STATUS    RESTARTS   AGE    IP           NODE     NOMINATED NODE   READINESS GATES
dnsutils                        1/1     Running   2          127m   10.244.2.8   node-2   <none>           <none>
nginx-deploy-7848d4b86f-4nvg7   1/1     Running   0          9m8s   10.244.2.9   node-2   <none>           <none>
nginx-deploy-7848d4b86f-prj7g   1/1     Running   0          9m8s   10.244.1.9   node-1   <none>           <none>
nginx-deploy-7848d4b86f-r95hq   1/1     Running   0          9m8s   10.244.1.8   node-1   <none>           <none>

The problem shows only when I tried to curl the nginx pods. It is not responsive.

root@master-node:~/k8s# curl 10.244.2.9
^C

I then login to the pod and confirmed that nginx is up.

root@master-node:~/k8s# kubectl exec -it nginx-deploy-7848d4b86f-4nvg7  -- /bin/bash
root@nginx-deploy-7848d4b86f-4nvg7:/# curl 127.0.0.1
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>
root@nginx-deploy-7848d4b86f-4nvg7:/# exit
exit

Here is the result of kubectl describe pod on one of the pods:

root@master-node:~/k8s# kubectl describe pod nginx-deploy-7848d4b86f-4nvg7
Name:         nginx-deploy-7848d4b86f-4nvg7
Namespace:    default
Priority:     0
Node:         node-2/192.168.108.12
Start Time:   Sun, 28 Mar 2021 04:49:15 +0000
Labels:       app=nginx
              pod-template-hash=7848d4b86f
Annotations:  <none>
Status:       Running
IP:           10.244.2.9
IPs:
  IP:           10.244.2.9
Controlled By:  ReplicaSet/nginx-deploy-7848d4b86f
Containers:
  nginx:
    Container ID:   docker://f6322e65cb98e54cc220a786ffb7c967bbc07d80fe8d118a19891678109680d8
    Image:          nginx
    Image ID:       docker-pullable://nginx@sha256:b0ea179ab61c789ce759dbe491cc534e293428ad232d00df83ce44bf86261179
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Sun, 28 Mar 2021 04:49:19 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xhkzx (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-xhkzx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-xhkzx
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  25m   default-scheduler  Successfully assigned default/nginx-deploy-7848d4b86f-4nvg7 to node-2
  Normal  Pulling    25m   kubelet            Pulling image "nginx"
  Normal  Pulled     25m   kubelet            Successfully pulled image "nginx" in 1.888247052s
  Normal  Created    25m   kubelet            Created container nginx
  Normal  Started    25m   kubelet            Started container nginx

I tried to troubleshoot by using: Debugging Kubernetes Networking

root@master-node:~/k8s# ip link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:db:6f:21 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:90:88:7c brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
    link/ether 02:42:1d:21:66:20 brd ff:ff:ff:ff:ff:ff
5: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
    link/ether 4a:df:fb:be:7b:0e brd ff:ff:ff:ff:ff:ff
6: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/ether 02:48:db:46:53:60 brd ff:ff:ff:ff:ff:ff
7: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether fa:29:13:98:2c:31 brd ff:ff:ff:ff:ff:ff
8: vethc2e0fa86@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default
    link/ether 7a:66:b0:97:db:81 brd ff:ff:ff:ff:ff:ff link-netnsid 0
9: veth3eb514e1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default
    link/ether 3e:3c:9d:20:5c:42 brd ff:ff:ff:ff:ff:ff link-netnsid 1
11: veth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 02:35:f0:fb:e3:b1 brd ff:ff:ff:ff:ff:ff link-netns test1
root@master-node:~/k8s# kubectl create -f nwtool-deployment.yaml
deployment.apps/nwtool-deploy created
root@master-node:~/k8s# kubectl get po
NAME                             READY   STATUS    RESTARTS   AGE
nwtool-deploy-6d8c99644b-fq6gv   1/1     Running   0          14s
nwtool-deploy-6d8c99644b-fwc6d   1/1     Running   0          14s
root@master-node:~/k8s# ^C
root@master-node:~/k8s# kubectl exec -it nwtool-deploy-6d8c99644b-fq6gv -- ip link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default
    link/ether 2e:02:b6:97:2f:10 brd ff:ff:ff:ff:ff:ff
root@master-node:~/k8s# kubectl exec -it nwtool-deploy-6d8c99644b-fwc6d -- ip link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
3: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default
    link/ether 82:21:fa:aa:34:27 brd ff:ff:ff:ff:ff:ff
root@master-node:~/k8s# ip link list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:db:6f:21 brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 08:00:27:90:88:7c brd ff:ff:ff:ff:ff:ff
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
    link/ether 02:42:1d:21:66:20 brd ff:ff:ff:ff:ff:ff
5: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default
    link/ether 4a:df:fb:be:7b:0e brd ff:ff:ff:ff:ff:ff
6: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/ether 02:48:db:46:53:60 brd ff:ff:ff:ff:ff:ff
7: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether fa:29:13:98:2c:31 brd ff:ff:ff:ff:ff:ff
8: vethc2e0fa86@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default
    link/ether 7a:66:b0:97:db:81 brd ff:ff:ff:ff:ff:ff link-netnsid 0
9: veth3eb514e1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP mode DEFAULT group default
    link/ether 3e:3c:9d:20:5c:42 brd ff:ff:ff:ff:ff:ff link-netnsid 1
11: veth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 02:35:f0:fb:e3:b1 brd ff:ff:ff:ff:ff:ff link-netns test1
root@master-node:~/k8s#

It looks that no veth pairs were created for the new pod on the master node. Any idea how to resolve this? Any help will be greatly appreciated. Thank you!

"kubectl get po -o wide" has been shown in my question. It is in the #3 code snip. — learning
Would it be possible that traffic is blocked among my VMs? How to troubleshoot further? All my VMs have both NAT and Host-Only Adapters. And they can ping each other. I recreated the environment on AWS and it is working fine. — learning
When I use kubectl exec to get into one of the nginx pod, I am able to curl the other nginx pod using flannel IP. But not from the flannel pods. — learning

learning learning · Accepted Answer · 2021-04-01T03:55:40

I have found out the issue. Thanks to: Kubernetes with Flannel — Understanding the Networking — Part 1 (Setup the demo) I have copied the excerpts that helped to resolve my issue below:

The VM’s will have 2 interfaces created. And, when running flannel, you would need to mention the interface name properly. Without that, you may see that the pods will come up and get the IP address, but can’t talk to each other.

You need to specify the interface name enp0s8 in flannel manifest file.

vagrant@master:~$ grep -A8 containers kube-flannel.yml
      containers:
      - name: kube-flannel
        image: quay.io/coreos/flannel:v0.10.0-amd64
        command:
        - /opt/bin/flanneld
        args:
        - --ip-masq
        - --kube-subnet-mgr
        - --iface=enp0s8          ####Add the iface name here.

If you happen to have different interfaces to be matched, you can match it on regex pattern. Let’s say the worker nodes could’ve enp0s8 or enp0s9 configured, then the flannel args would be — --iface-regex=[enp0s8|enp0s9]

Kubernetes: Unable to access flannel pod

1 Answers