Kubernetes DNS not resolving external addresses

Question

Having some issue with my kubernetes cluster and DNS.

We recently updated to RHEL 7.5 and one of the machines was the Master. Once it back online most everything worked but I just noticed that external connections from the cluster do not resolve. Internal communication works great.

Here's the busybox nslookup results:

kubectl exec -ti busybox -- nslookup kubernetes.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

nslookup: can't resolve 'kubernetes.default'

kubedns status

Name:              kube-dns
Namespace:         kube-system
Labels:            k8s-app=kube-dns
                   kubernetes.io/cluster-service=true
                   kubernetes.io/name=KubeDNS
Annotations:       <none>
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                10.96.0.10
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         10.32.0.18:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         10.32.0.18:53
Session Affinity:  None
Events:            <none>

All pods say they are up

NAME                        READY     STATUS    RESTARTS   AGE
kube-dns-86f4d74b45-9m292   3/3       Running   26         44d

Lastest logs from the kubedns pods

I0523 15:59:14.291623       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0523 15:59:14.291638       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0523 15:59:14.791440       1 dns.go:170] Initialized services and endpoints from apiserver
I0523 15:59:14.791560       1 server.go:135] Setting up Healthz Handler (/readiness)
I0523 15:59:14.791579       1 server.go:140] Setting up cache handler (/cache)
I0523 15:59:14.791588       1 server.go:126] Status HTTP port 8081
I0523 16:48:05.175159       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-controller-manager" in namespace "kube-system". DNS records will be created once endpoints show up.
I0523 16:48:05.236418       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-dns" in namespace "kube-system". DNS records will be created once endpoints show up.
I0523 16:48:05.252139       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-etcd" in namespace "kube-system". DNS records will be created once endpoints show up.
I0523 16:48:05.264292       1 dns.go:555] Could not find endpoints for service "kube-prometheus-exporter-kube-scheduler" in namespace "kube-system". DNS records will be created once endpoints show up.
I0524 13:04:38.008038       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:48431->10.101.24.100:53: i/o timeout"
I0524 13:04:41.973900       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:45738->10.101.24.100:53: i/o timeout"
I0524 17:26:54.056913       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:52242->10.101.24.100:53: i/o timeout"
I0524 18:28:16.762002       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:43051->10.101.24.100:53: i/o timeout"
I0524 18:28:20.762912       1 logs.go:41] skydns: failure to forward request "read udp 10.32.0.18:57624->10.101.24.100:53: i/o timeout"

I am assuming this has something to do with the reboot.

Kubernetes: version 1.10.3

DNS: kube-dns

Network: Weavenet

Host OS: RHEL 7.5

Any help would be great.

Thanks

erstaples erstaples · Accepted Answer · 2018-05-25T00:31:19

Not sure of the details of your upgrade, but check to see if your kube-dns version changed when you upgraded your OS. There’s an issue with kube-dns version 1.14.9 resolving external name services. You’ll need to roll back kube-dns to a previous build, or upgrade to 1.14.10.

See this issue: https://github.com/kubernetes/kops/issues/4986

Kubernetes DNS not resolving external addresses

1 Answers