External dns lookup fails in pods

Question

I spinned up k8 cluster using Kesley KTHW (https://github.com/kelseyhightower/kubernetes-the-hard-way/) in GCP.

Trying to do some exercise on this link => https://github.com/dgkanatsios/CKAD-exercises/blob/master/b.multi_container_pods.md and my external dns resolution fails from the pod.

Version:

sshanmugagani@MSI:~/cka/skk8/practise-1$ kubectl version Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:58:53Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.6", GitCommit:"dff82dc0de47299ab66c83c626e08b245ab19037", GitTreeState:"clean", BuildDate:"2020-07-15T16:51:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Pod fails to resolve google.com:

sshanmugagani@MSI:~/cka/skk8/practise-1$ kubectl exec -ti dnsutils -- nslookup google
Server: 10.32.0.10
Address: 10.32.0.10#53

** server can't find google.us-west1-c.c.test.internal: SERVFAIL

command terminated with exit code 1

Pod's /etc/resolv.conf:

sshanmugagani@MSI:~/cka/skk8/practise-1$ kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml pod/dnsutils created

sshanmugagani@MSI:~/cka/skk8/practise-1$ k exec -it dnsutils -- cat /etc/resolv.conf search default.svc.cluster.local svc.cluster.local cluster.local us-west1-c.c.test.internal c.test.internal google.internal nameserver 10.32.0.10 options ndots:5

Getting worker node where pod runs:

sshanmugagani@MSI:~/cka/skk8/practise-1$ kgp -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES dnsutils 1/1 Running 0 60s 10.200.0.65 worker-0 <none> <none> multi 0/2 Completed 0 12h 10.200.0.53 worker-0 <none> <none>

Worker node resolves:

sshanmugagani@worker-0:~$ nslookup google.com
Server: 127.0.0.53
Address: 127.0.0.53#53

Non-authoritative answer:
Name: google.com
Address: 74.125.20.101
Name: google.com
Address: 74.125.20.100
Name: google.com
Address: 74.125.20.139
Name: google.com
Address: 74.125.20.138
Name: google.com
Address: 74.125.20.113
Name: google.com
Address: 74.125.20.102
Name: google.com
Address: 2607:f8b0:400e:c09::65
Name: google.com
Address: 2607:f8b0:400e:c09::8a
Name: google.com
Address: 2607:f8b0:400e:c09::8b
Name: google.com
Address: 2607:f8b0:400e:c09::71

Coredns:

sshanmugagani@MSI:~/cka/skk8/practise-1$ kgp $ks NAME READY STATUS RESTARTS AGE coredns-5677dc4cdb-cfl2j 1/1 Running 1 11h coredns-5677dc4cdb-xqm44 1/1 Running 1 11h

Coredns logs:

sshanmugagani@MSI:~/cka/skk8/practise-1$ kubectl logs coredns-5677dc4cdb-cfl2j $ks .:53 [INFO] plugin/reload: Running configuration MD5 = fbb756dad13bce75afc40db627b38529 CoreDNS-1.7.0 linux/amd64, go1.14.4, f59c03d [ERROR] plugin/errors: 2 2953017454530458158.338294255644342916. HINFO: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 google.com.us-west1-c.c.test.internal. A: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 google.com. A: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 google.com. A: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 kube-dns. A: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com.us-west1-c.c.test.internal. AAAA: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com.c.test.internal. AAAA: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com. AAAA: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com.c.test.internal. AAAA: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com.goo. AAAA: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com.us-west1-c.c.test.internal. AAAA: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com.c.test.internal. AAAA: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com.goo. AAAA: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com. A: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com.goo. A: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com. A: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com.us-west1-c.c.test.internal. A: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com.c.test.internal. A: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com.goo. A: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 neverssl.com.us-west1-c.c.test.internal. A: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 google.com. A: plugin/loop: no next plugin found [ERROR] plugin/errors: 2 google. A: plugin/loop: no next plugin found

You need to use google.com instead only google. This is the reason dns is trying to find a internal pod named google. You did it right in the nod, but not in the dnsutil command. Or it was a misspelling? — Mr.KoopaKiller
Hello @KoopaKiller - thanks for the reply. I did used the FQDN but didn't posted that output.Got another suggestion on adding a forward to /etc/resolv.conf as below,. Now am stuck with another error as shown below, apiVersion: v1 kind: ConfigMap metadata: name: coredns namespace: kube-system data: Corefile: | .:53 { errors health ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } prometheus :9153 forward . /etc/resolv.conf cache 30 loop reload loadbalance } — sshanmugagani
@KoopaKiller - Now I had end up in state where 'dig` command works but nslookup, wget still fails as shown below. No errors though in other svcs. sshanmugagani@MSI:~/cka/skk8/k8s-the-hard-way-script$ k exec -it dnsutils -- /bin/sh / # wget -O /work-dir/index.html http://neverssl.com/online wget: bad address 'neverssl.com' / # wget -O /work-dir/index.html http://neverssl.com/online wget: bad address 'neverssl.com' / # nslookup neverssl.com ;; reply from unexpected source: 10.200.1.8#53, expected 10.32.0.10#53 ;; reply from unexpected source: 10.200.1.8#53, expected 10.32.0.10#53 ^C — sshanmugagani

sshanmugagani sshanmugagani · Accepted Answer · 2020-11-27T15:04:54

Got this issue fixed by ==> Loaded kernel module modprobe br_netfilter ===> Add flag for kube-proxy start up file --masquerade-all

External dns lookup fails in pods

1 Answers