DNS lookup not working properly in Kubernetes cluster

Question

I spin up a cluster with minikube then apply this dummy deployment/service

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      run: nginx-label
  replicas: 2
  template:
    metadata:
      labels:
        run: nginx-label
    spec:
      containers:
      - name: nginx-container
        image: nginx
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-service
  labels:
    run: nginx-label
spec:
  ports:
  - port: 1234
    targetPort: 80
    protocol: TCP
  selector:
    run: nginx-label

Then I create a dummy curl pod to test internal network with the following

kubectl run curl --image=radial/busyboxplus:curl -i --tty

Inside that curl instance, I'm able to access the nginx with $NGINX_SERVICE_SERVICE_HOST:$NGINX_SERVICE_SERVICE_PORT or nginx-service.default:1234, but not nginx-service:1234, even though those pods belong to the same namespace.

ubuntu:~$ kubectl get pods --namespace=default
NAME                                READY   STATUS    RESTARTS   AGE
curl-69c656fd45-d7w8t               1/1     Running   1          29m
nginx-deployment-58595d65fc-9ln25   1/1     Running   0          29m
nginx-deployment-58595d65fc-znkqp   1/1     Running   0          29m

Any idea what could cause this? Following is the nslookup result

[ root@curl-69c656fd45-d7w8t:/ ]$ nslookup nginx-service
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nginx-service
Address 1: 23.202.231.169 a23-202-231-169.deploy.static.akamaitechnologies.com
Address 2: 23.217.138.110 a23-217-138-110.deploy.static.akamaitechnologies.com
[ root@curl-69c656fd45-d7w8t:/ ]$ nslookup nginx-service.default
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      nginx-service.default
Address 1: 10.103.69.73 nginx-service.default.svc.cluster.local
[ root@curl-69c656fd45-d7w8t:/ ]$

Update: here's the content of /etc/resolv.conf

[ root@curl-69c656fd45-d7w8t:/ ]$ cat /etc/resolv.conf 
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local attlocal.net
options ndots:5
[ root@curl-69c656fd45-d7w8t:/ ]$

``` [ root@curl-69c656fd45-d7w8t:/ ]$ cat /etc/resolv.conf nameserver 10.96.0.10 search default.svc.cluster.local svc.cluster.local cluster.local attlocal.net options ndots:5 [ root@curl-69c656fd45-d7w8t:/ ]$ ``` — Hanks
Can you paste the output in proper format and edit question with this information ? — Malgorzata

Malgorzata Malgorzata · Accepted Answer · 2020-04-21T11:01:42

Answering on your question in comment. Check following lines below the field name in nslookup file, for host nginx-service they are :

Name:      nginx-service
Address 1: 23.202.231.169 a23-202-231-169.deploy.static.akamaitechnologies.com
Address 2: 23.217.138.110 a23-217-138-110.deploy.static.akamaitechnologies.com

for nginx-service.default:

Name:      nginx-service.default
Address 1: 10.103.69.73 nginx-service.default.svc.cluster.local

The flow is:

Executing command nslookup
Checking entrypoints(line below host with adress and hostname)
Comparing entrypoints and its domains to domain listed in /etc/resolv.conf file if it doesn't match this mean that we cannot reach specific host.

Your nginx-service"hit" a23-202-231-169.deploy.static.akamaitechnologies.com NOTnginx-service.default.svc.cluster.local.
nginx-service.default "hit" nginxservice.default.svc.cluster.local as you said that is why curl is working.

10.96.0.10 is the address of our system's Domain Name Server named kube-dns.kube-system.svc.cluster.local. This is the server your system is configured to use to translate domain names into IP addresses.

Speaking about the domain deploy.static.akamaitechnologies.com Akamai is a Content Delivery Network used by Symantec (and many other companies). Kubernetes is a cloud computing service used by a large number of companies as well for internet content. These services are essential for providing content when you visit certain websites, and for delivering things like product updates. It is normal to see connections to these sites when you visit certain other websites or have certain products, like Norton, installed. They essentially provide the servers needed to propagate large amounts of data to various regions of the world quickly while balancing internet traffic so that individual server locations are not overloaded..

Hmm, sorry, not really...My question is that, based on the search directive in /etc/resolv.conf, if nginx-service.default is resolved to nginx-service.default.svc.cluster.local, so should nginx-service. Did I miss anything?

Answer:

Keep in mind that:

Look firstly at nslookup
And then find match in /etc/resolv.conf

Note: In case you will do these steps vice versa, it won't work.

Reasons what may be the problem of wrong resolution of domain in nslookup are many - see: dns-debugging . Try to execute command dig nginx-service. Then interpret the output from this file to find the real problem. Because it is obvious that you cannot curl nginx-service (I have explained it above) but why nslookup shows different records for nameservers it is completely different question.

More information you can find here: nslookup, akamai.

DNS lookup not working properly in Kubernetes cluster

1 Answers