Kubernetes service discovery doesn't resolve service host on minikube

Question

I'm trying to run two services with k8s running on minikube installation. So I have the sevice Foo and Bar, so when I'm accessing the service Foo it must trigger service Bar to get the data.

Here is the deployment configuration for Foo service foo-deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: foo-server
  labels:
    app: foo-server
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: foo-server
        serving: "true"
    spec:
      containers:
      - name: foo-server
        image: foo-container
        env:
        - name: BAR_SERVICE_URL
          value: http://bar-server:8081
        ports:
        - containerPort: 8080
          name: http

foo-svc.yaml:

apiVersion: v1
kind: Service
metadata:
  name: foo-server
  labels:
    app: foo-server
spec:
  type: NodePort
  ports:
  - port: 8080
    targetPort: 8080
    name: http
  selector:
    app: foo-server
    serving: "true"

The BAR_SERVICE_URL evn variable is used inside the Foo so the service knows the host to Bar. And here is the service configuration for Bar service bar-svc.yaml:

apiVersion: v1
kind: Service
metadata:
  name: bar-server
  labels:
    app: bar-server
spec:
  type: NodePort
  ports:
  - port: 8081
    targetPort: 8081
    name: http
  selector:
    app: bar-server

bar-deployment.yaml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: bar-server-v1
  labels:
    app: bar-server
    version: "1.0"
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: bar-server
        version: "1.0"
        serving: "true"
    spec:
      containers:
      - name: bar-server
        image: bar-container
        ports:
        - containerPort: 8081
          name: http

When I'm accessing the Foo service with the minikube ip and exposed service port I receive the following error in the pod logs:

2018-02-08 14:32:25.875 ERROR 1 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://bar-server:8081/endpoint": bar-server; nested exception is java.net.UnknownHostException: bar-server] with root cause

java.net.UnknownHostException: bar-server

However, I can access the bar-server/endpoint using the minikube ip and exposed port like this http://192.168.99.100:31168/endpoint.

So looks like I can access the Foo and Bar services from outside the cluster. But Foo service cannot resolve the Bar service host and cannot access it.

UPD: Here are some update with troubleshouting details.

On minikube the kube-dns addon is enabled.
Trying from the iside of Foo pod to wget to the Bar service endpoint. I'm able to do it via IP address of the bar-server but not with bar-server:8081/endpoint.
Trying nslookup kubernetes.default.svc.cluster.local localhost looks good. Here is the ouptput:

Server: 127.0.0.1 Address 1: 127.0.0.1 localhost
Also dns pod looks up and running. Here is the output for kubectl get pods -n=kube-system | grep -i dns

kube-dns-6777479f6b-vxd7m 3/3 Running 9 1d

Hi Sergii, were you able to solve this? I have a similar issue on my own Minikube deployment. — alexvicegrab
@alexvicegrab my case was that I have update the minikube without doing minikube delete so it was fixed by deleting the minikube cluster and starting it again — Sergii Bishyr
Thanks for the quick answer! In my own case there seems to be some issue with how the Peer interacts with services on K8S when using Minikube. Same thing works completely fine when I use a real K8S cluster on the cloud. — alexvicegrab

Vikram Hosakote Vikram Hosakote · Accepted Answer · 2018-02-09T07:01:55

Kubernetes does service discovery using DNS (kube-dns). If DNS in k8s is working, service Foo should be able to ping or curl service Bar, and vice-versa.

A way to check if kube-dns is working is to see if the following commands work inside any pod:

nslookup kubernetes.default.svc.cluster.local localhost
nslookup kubernetes.default localhost

kube-dns pod's health can be checked in the output of:

kubectl get pods -n=kube-system | grep -i dns

Kubernetes service discovery doesn't resolve service host on minikube

2 Answers