Kube-dns service discovery cannot discover port number of service

Question

I am using DNS based service discovery to discover services in K8s cluster. From this link it is very clear that to discover a service named my-service we can do name lookup "my-service.my-ns" and the pod should be able to find the services.

However in case of port discovery for the service the solution is to use is "_http._tcp.my-service.my-ns" where

_http refers to the port named http in my-service.

But even after using _http._tcp.my-service it doesn't resolve port number. Below are the details.

my-service which needs to be discovered

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-service
  ports:
    - name: http
      protocol: TCP
      port: 5000
      targetPort: 5000

client-service yaml snippet trying to discover my-service and its port.

spec:
  containers:
  - name: client-service
    image: client-service
    imagePullPolicy: Always
    ports:
      - containerPort: 7799
    resources:
      limits:
        cpu: "100m"
        memory: "500Mi"
    env:
    - name: HOST
      value: my-service
    - name: PORT
      value: _http._tcp.my-service

Now when I make a request it fails and logs following request which is clearly incorrect as it doesn't discover port number.

 http://my-service:_http._tcp.my-service

I am not sure what wrong I am doing here, but I am following the same instructions mentioned in document.

Can somebody suggest what is wrong here and how we can discover port using DNS based service discovery? Is my understanding wrong here that it will return the literal value of port?

Cluster details

K8s cluster version is 1.11.5-gke.5 and Kube-dns is running

Additional details trying to discover service from busybox and its not able to discover port value 5000

kubectl exec  busybox -- nslookup my-service
Server:         10.51.240.10
Address:        10.51.240.10:53

Name:   my-service.default.svc.cluster.local
Address: 10.51.253.236

*** Can't find my-service.svc.cluster.local: No answer
*** Can't find my-service.cluster.local: No answer
*** Can't find my-service.us-east4-a.c.gdic-infinity-dev.internal: No answer
*** Can't find my-service.c.gdic-infinity-dev.internal: No answer
*** Can't find my-service.google.internal: No answer
*** Can't find my-service.default.svc.cluster.local: No answer
*** Can't find my-service.svc.cluster.local: No answer
*** Can't find my-service.cluster.local: No answer
*** Can't find my-service.us-east4-a.c.gdic-infinity-dev.internal: No answer
*** Can't find my-service.c.gdic-infinity-dev.internal: No answer
*** Can't find my-service.google.internal: No answer


kubectl exec  busybox -- nslookup _http._tcp.my-service
Server:         10.51.240.10
Address:        10.51.240.10:53

** server can't find _http._tcp.my-service: NXDOMAIN

*** Can't find _http._tcp.my-service: No answer

kubectl exec -it busybox /bin/bash | nslookup my-service for sure does not do what you want; (a) busybox has no bash (b) that command is running nslookup on your machine and not the busybox Pod (c) that syntax is absolutely crazy, as you are piping the output of bash into the nslookup command — mdaniel
Thanks @MatthewLDaniel for pointing out the mistake. Originially I tried with right syntax but during composing question somehow I made this mistake. I have corrected it now. The still busybox example I mentioned was for additional information to eliminate any speculation around bad yaml config possibility, Can you pls undo your downvote now? I think its valid contents now. — Neeraj

David Maze David Maze · Accepted Answer · 2019-01-20T23:36:20

Since Services come with their own (Kubernetes-internal) IP addresses, the easy answer here is to not pick arbitrary ports for Services. Change to port: 80 in your Service definition, and clients will be able to reach it using the default HTTP port. When you set the environment variable, set

- name: PORT
  value: "80"

DNS supports several different record types; for example, an A record translates a host name to its IPv4 address, and AAAA to an IPv6 address. The Kubernetes Service documentation you cite notes (emphasis mine)

you can do a DNS SRV query ... to discover the port number for "http".

While SRV records seems like they solve both halves of this problem (they provide a port and a host name for a service) in practice they seem to get fairly little use. The linked Wikipedia page has a list of services that use it, but "connect to the thing this SRV record points at" isn't an option in mainstream TCP clients that I know of.

You should be able to verify this with a command like (running this debugging image)

kubectl run debug --rm -it --image giantswarm/tiny-tools sh
# dig -t srv _http._tcp.my-service

(But notice the -t srv argument; it is not the default record type.)

Most things that expect a PORT environment variable or similar expect a number, or if not, a name they can find in an /etc/services file. The syntax you're trying to use here and trying to provide a DNS SRV name instead probably just won't work, unless you know the specific software supports it.

Kube-dns service discovery cannot discover port number of service

1 Answers