External DNS resolution stopped working in Container Engine

Question

I have a simple container on Google Container Engine that has been running for months with no issues. Suddenly, I cannot resolve ANY external domain. In troubleshooting I have re-created the container many times, and upgraded the cluster version to 1.4.7 in an attempt to resolve with no change.

To rule the app code out as much as possible, even a basic node.js code cannot resolve an external domain:

const dns = require('dns');
dns.lookup('nodejs.org', function(err, addresses, family) {
    console.log('addresses:', addresses);
});
/* logs 'undefined' */

The same ran on a local machine or local docker container works as expected.

This kubectl call fails as well:

# kubectl exec -ti busybox -- nslookup kubernetes.default
nslookup: can't resolve 'kubernetes.default'

Two show up when getting kube-dns pods (admittedly not sure if that is expected)

# kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME                 READY     STATUS    RESTARTS   AGE
kube-dns-v20-v8pd6   3/3       Running   0          1h
kube-dns-v20-vtz4o   3/3       Running   0          1h

Both say this when trying to check for errors in the DNS pod:

# kubectl logs --namespace=kube-system pod/kube-dns-v20-v8pd6 -c kube-dns
Error from server: container kube-dns is not valid for pod kube-dns-v20-v8pd6

I expect the internally created kube-dns is not properly pulling external DNS results or some other linkage disappeared.

I'll accept almost any workaround if one exists, as this is a production app - perhaps it is possible to manually set nameservers in the Kubernetes controller YAML file or elsewhere. Setting the contents of /etc/resolv.conf in Dockerfile does not seem to work.

Christiaan Christiaan · Accepted Answer · 2017-01-24T11:19:53

Just checked and in our own clusters we usually have 3 kube-dns pods so something seems off there.

What does this say: kybectl describe rc kube-dns-v20 --namespace=kube-system

What happens when you kill the kube-dns pods? (the rc should automatically restart them)

What happens when you do an nslookup with a specific nameserver? nslookup nodejs.org 8.8.8.8

External DNS resolution stopped working in Container Engine

1 Answers