Google GKE kubernetes DNS fail to resolve service name

Question

I am having an issue with Kubernetes on GKE. I am unable to resolve services by their name using internal DNS.

this is my configuration

Google GKE v1.15

kubectl get namespaces
NAME              STATUS   AGE
custom-metrics    Active   183d
default           Active   245d
dev               Active   245d
kube-node-lease   Active   65d
kube-public       Active   245d
kube-system       Active   245d
stackdriver       Active   198d

I've deployed a couple of simple services based on openjdk 11 docker image and made with spring boot + actuator in order to have a /actuator/health endpoint to test in dev

kubectl get pods --namespace=dev
NAME                          READY   STATUS    RESTARTS   AGE
test1-5d86946c49-h9t9l        1/1     Running   0          3h1m
test2-5bb5f4ff8d-7mzc8        1/1     Running   0          3h10m

If i try to execute under

kubectl --namespace=dev exec -it test1-5d86946c49-h9t9  -- /bin/bash
root@test1-5d86946c49-h9t9:/app# cat /etc/resolv.conf
nameserver 10.40.0.10
search dev.svc.cluster.local svc.cluster.local cluster.local europe-west1-b.c.back-office-236415.internal c.back-office-236415.internal google.internal
options ndots:5
root@test1-5d86946c49-h9t9:/app# nslookup test2
Server:         10.40.0.10
Address:        10.40.0.10#53

** server can't find test2: NXDOMAIN

The same issue occurs if I try using test2 service and try to resolve test1. There is a special configuration for namespace to enable DNS resolve? Shouldn't this be automatic?

Have Kubernetes Services named test1 and test2 been created for the test1 and test2 (naked) Pods/Deployments, respectively? E.g. does this return any Kubernetes Services: kubectl get svc --namespace=dev? — gears
yes both services have been created with a Deployment, but you comment was very helpfull, after i've created a couple of services (nodeport) i was able to query test1 and test2 throudht nodeports services. I thought that pods were not necessary because kuberenetes automatically exposes each pod thought a DNS name. Thanks a lot — IlTera

Shanewaz Shanewaz · Accepted Answer · 2019-12-21T00:02:34

I have reproduced this using master version 1.15 and and type of service as ‘ClusterIP’. I am able to do look up from the Pod of one service to another. For creating Kubernetes Services in a Google Kubernetes Engine cluster [1] might be helpful.

To see the services: $ kubectl get svc --namespace=default

To access the deployment: $ kubectl exec -it [Pod Name] sh

To lookup: $ nslookup [Service Name]

Every Service defined in the cluster (including the DNS server itself) is assigned a DNS name. By default, a client Pod’s DNS search list will include the Pod’s own namespace and the cluster’s default domain.

“Normal” (not headless) Services are assigned a DNS A record for a name of the form my-svc.my-namespace.svc.cluster-domain.example. This resolves to the cluster IP of the Service.

For “Headless” (without a cluster IP) Services are also assigned a DNS A record for a name.Though this resolves to the set of IPs of the pods selected by the Service.

However, DNS policies can be set on a per-pod basis. Currently Kubernetes supports the following pod-specific DNS policies. These policies are specified in the dnsPolicy field of a Pod Spec [2]:

“Default“: The Pod inherits the name resolution configuration from the node that the pods run on.

“ClusterFirst“: Any DNS query that does not match the configured cluster domain suffix, such as “www.kubernetes.io”, is forwarded to the upstream nameserver inherited from the node. Cluster administrators may have extra stub-domain and upstream DNS servers configured.

“ClusterFirstWithHostNet“: For Pods running with hostNetwork, need to set its DNS policy “ClusterFirstWithHostNet”.

“None“: It allows a Pod to ignore DNS settings from the Kubernetes environment. All DNS settings are supposed to be provided using the dnsConfig field in the Pod Spec.

[1]-https://cloud.google.com/kubernetes-engine/docs/how-to/exposing-apps [2]-https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-config

Google GKE kubernetes DNS fail to resolve service name

1 Answers