kubernetes service discovery with specific endpoint (DNS)

Question

I am using AKS cluster on Azure. I am trying to discover service using DNS (http://my-api.default.svc.cluster.local:3000/) but, it's not working (This site can’t be reached). With service IP endpoint everything is working fine.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-api
  labels:
    app: my-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-api
  template:
    metadata:
      labels:
    app: my-api
    spec:
      containers:
      - name: my-api
    image: test.azurecr.io/my-api:latest
    ports:
    - containerPort: 3000
      imagePullSecrets:
      - name: testsecret
---
apiVersion: v1
kind: Service
metadata:
  name: my-api
spec:
  selector:
    app: my-api
  ports:
  - protocol: TCP
    port: 3000
    targetPort: 3000

kubectl describe services kube-dns --namespace kube-system

Name:              kube-dns
Namespace:         kube-system
Labels:            addonmanager.kubernetes.io/mode=Reconcile
               k8s-app=kube-dns
               kubernetes.io/cluster-service=true
               kubernetes.io/name=KubeDNS
Annotations:       kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"addonmanager.kubernetes.io/mode":"Reconcile","k8s-app":"kube-dns","kubernet...
Selector:          k8s-app=kube-dns
Type:              ClusterIP
IP:                10.10.110.110
Port:              dns  53/UDP
TargetPort:        53/UDP
Endpoints:         10.10.100.54:53,10.10.100.64:53
Port:              dns-tcp  53/TCP
TargetPort:        53/TCP
Endpoints:         10.10.100.54:53,10.10.100.64:53
Session Affinity:  None
Events:            <none>

kubectl describe svc my-api

Name:              my-api
Namespace:         default
Labels:            <none>
Annotations:       kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"my-api","namespace":"default"},"spec":{"ports":[{"port":3000,"protocol":...
Selector:          app=my-api
Type:              ClusterIP
IP:                10.10.110.104
Port:              <unset>  3000/TCP
TargetPort:        3000/TCP
Endpoints:         10.10.100.42:3000
Session Affinity:  None
Events:            <none>

From Second POD

kubectl exec -it second-pod /bin/bash
curl my-api.default.svc.cluster.local:3000
Response: {"value":"Hello world2"}

From Second POD website is running which is using the same endpoint but it's not connecting to the service.

Answer 1

Fixing the indentation of your yaml file, I was able to launch the deployment and service successfully. Also the DNS resolution worked fine.

Differences:

• Fixed indentation
• Used test1 namespaces instead of default
• Used containerPort 80 instead of 3000
• Used my image

Deployment:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  labels:
    app: my-api
  name: my-api
  namespace: test1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: my-api
  template:
    metadata:
      labels:
        app: my-api
    spec:
      containers:
      - image: leodotcloud/swiss-army-knife
        name: my-api
        ports:
        - containerPort: 80
          protocol: TCP

Service:

apiVersion: v1
kind: Service
metadata:
  name: my-api
  namespace: test1
spec:
  ports:
  - port: 3000
    protocol: TCP
    targetPort: 80
  selector:
    app: my-api
  type: ClusterIP

Debugging steps:

• Install tcpdump inside both of the kube-dns containers and start capturing DNS traffic (with filters from the second pod IP)
• From inside the second pod, run curl or dig command using the FQDN.
• Check if the DNS query packets are reaching the kube-dns containers.
• If not, check for networking issues.
• If the DNS resolution is working, then start tcpdump inside your application container and check if the curl packet is reaching the container.
• Check the source and destination IP address of the packets.
• Check the iptables rules on the hosts.
• Check sysctl settings.

Answer 2

If you use Deployment to deploy your application onto cluster where it will be consumed via a Service you should have no need at all to manually set Endpoints. Just rely on kubernetes and define normal selector in your Service object.

Other then that, when it makes sense (external service consumed from within cluster), you need to make sure your Endpoints ports definition fully matches the one on service (incl. protocol and potentially name). This incomplete matching is a most common reason for endpoints to be not visible as a part of service.

Answer 3

From the above discussion, what I understood is, you want to expose a service but not using the IP address. Service can be exposed in many ways. you should look for Service type LoadBalancer.

Try modifying your service is follow :

apiVersion: v1
kind: Service
metadata:
  name: my-api
spec:
  type: LoadBalancer
  selector:
    app: my-api
  ports:
  - protocol: TCP
    port: 3000
    targetPort: 3000

This will create a loadbalancer and map your service to the same.

Later you can add this loadbalancer to your DNS mapping service provided by Azure to give the domain name you like. ex: http:\\my-api.example.com:3000

Also I would like to add, if you define your ports as follow :

ports:
  - name: http
    port: 80
    targetPort: 3000

This will redirect traffic coming to port 80 to 3000 and your service call would look much cleaner for ex. http:\\my-api.example.com