Kubernetes DNS not working on local Ubuntu 18.04 environment

Question

I trying to deploy Kubernetes system on my local computer (Ubuntu 18.04) and have some problems with the DNS service (I can't talk with headless services through their DNS name).

I use minikube for running the cluster and the version is -

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:30:10Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.2", GitCommit:"59603c6e503c87169aea6106f57b9f242f64df89", GitTreeState:"clean", BuildDate:"2020-01-18T23:22:30Z", GoVersion:"go1.13.5", Compiler:"gc", Platform:"linux/amd64"}

The headless service -

NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
kubernetes           ClusterIP   10.96.0.1    <none>        443/TCP             67m
zookeeper-headless   ClusterIP   None         <none>        2888/TCP,3888/TCP   3m58s

The pods -

NAME                      READY   STATUS    RESTARTS   AGE
zookeeper-statefulset-0   1/1     Running   1          57m
zookeeper-statefulset-1   1/1     Running   1          56m
zookeeper-statefulset-2   1/1     Running   1          54m

DNS service endpoint which not exists - (kubectl get ep kube-dns --namespace=kube-system)

NAME       ENDPOINTS   AGE
kube-dns               68m

The DNS pods (not ready) -

NAME                       READY   STATUS    RESTARTS   AGE
coredns-6955765f44-gv42p   0/1     Running   1          58m
coredns-6955765f44-rfkm2   0/1     Running   1          58m

and the logs of DNS pod is -

[INFO] plugin/reload: Running configuration MD5 = 4e235fcc3696966e76816bcd9034ebc7 CoreDNS-1.6.5 linux/amd64, go1.13.4, c2fd1b2 [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" E0221 12:50:23.090626 1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090668 1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090671 1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout I0221 12:50:23.090594 1 trace.go:82] Trace[146678255]: "Reflector pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98 ListAndWatch" (started: 2020-02-21 12:49:53.090061147 +0000 UTC m=+0.011664556) (total time: 30.000405618s): Trace[146678255]: [30.000405618s] [30.000405618s] END E0221 12:50:23.090626 1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090626 1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090626 1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Endpoints: Get https://10.96.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout I0221 12:50:23.090644 1 trace.go:82] Trace[653875127]: "Reflector pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98 ListAndWatch" (started: 2020-02-21 12:49:53.090057185 +0000 UTC m=+0.011660587) (total time: 30.00054106s): Trace[653875127]: [30.00054106s] [30.00054106s] END I0221 12:50:23.090654 1 trace.go:82] Trace[1501712764]: "Reflector pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98 ListAndWatch" (started: 2020-02-21 12:49:53.090052023 +0000 UTC m=+0.011655434) (total time: 30.000437703s): Trace[1501712764]: [30.000437703s] [30.000437703s] END E0221 12:50:23.090668 1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090668 1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090668 1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090671 1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090671 1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout E0221 12:50:23.090671 1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: i/o timeout [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes" [INFO] plugin/ready: Still waiting on: "kubernetes"

I tried running -

kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm
/ # nslookup headless.default.svc.cluster.local

and got -

Server:    10.96.0.10
Address 1: 10.96.0.10

nslookup: can't resolve 'headless.default.svc.cluster.local'

I don't even know where to start to fix that. Can anyone help?

UPDATE

I think I understand what is causing the problem but I do not understand why it is happening. The problem seems to be happening after the FireWall is activated. For some reason, the core-dns pods can't run and get stuck in a ready state. After I turned off the FireWall by running - sudo ufw disable The core-dns pods state has changed to Running and the service now has endpoint addresses.

 kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm
If you don't see a command prompt, try pressing enter.
/ # nslookup  zookeeper-headless.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      zookeeper-headless.default
Address 1: 172.17.0.4 zookeeper-statefulset-1.zookeeper-headless.default.svc.cluster.local
Address 2: 172.17.0.5 zookeeper-statefulset-0.zookeeper-headless.default.svc.cluster.local
Address 3: 172.17.0.6 zookeeper-statefulset-2.zookeeper-headless.default.svc.cluster.local

.

NAME       ENDPOINTS                                               AGE
kube-dns   172.17.0.2:53,172.17.0.3:53,172.17.0.2:53 + 3 more...   34m

.

NAMESPACE     NAME                             READY   STATUS    RESTARTS           12m
kube-system   coredns-6955765f44-2d8md         1/1     Running   4          34m
kube-system   coredns-6955765f44-n2gcp         1/1     Running   4          34m

No logs, you just copy/paste the output of kubectl describe of the pod. — Kartoch
what cni are you using ? flannel or something else? What is the pod cidr you have given? How did you setup the cluster, kubeadm or something else? — Arghya Sadhu
You're trying to reach headless.default.svc.cluster.local but I can't see any service with this name in your services list. Please try reaching zookeeper-headless with $ nslookup zookeeper-headless.default.svc.cluster.local — Mark Watney

nirkov nirkov · Accepted Answer · 2020-02-21T14:22:05

I'm thinking I understand what is causing the problem but I do not understand why it is happening. The problem seems to be happening after the FireWall is activated. For some reason, the core-dns pods can't run and get stuck in a ready state. After I turned off the FireWall by running -

sudo ufw disable

The core-dns pods state has changed to Running and the service now has endpoint addresses.

 kubectl run -i --tty --image busybox:1.28 dns-test --restart=Never --rm
If you don't see a command prompt, try pressing enter.
/ # nslookup  zookeeper-headless.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      zookeeper-headless.default
Address 1: 172.17.0.4 zookeeper-statefulset-1.zookeeper-headless.default.svc.cluster.local
Address 2: 172.17.0.5 zookeeper-statefulset-0.zookeeper-headless.default.svc.cluster.local
Address 3: 172.17.0.6 zookeeper-statefulset-2.zookeeper-headless.default.svc.cluster.local

.

NAME       ENDPOINTS                                               AGE
kube-dns   172.17.0.2:53,172.17.0.3:53,172.17.0.2:53 + 3 more...   34m

.

NAMESPACE     NAME                             READY   STATUS    RESTARTS           12m
kube-system   coredns-6955765f44-2d8md         1/1     Running   4          34m
kube-system   coredns-6955765f44-n2gcp         1/1     Running   4          34m

Kubernetes DNS not working on local Ubuntu 18.04 environment

1 Answers