Deployment pod cannot connect ClusterIP service

1
votes

I try to expose my server IP by using Ingress.

The server is an Express.js app. It listens at http://localhost:5000 locally when without docker.

Here are my Kubernetes config files:

server-deployment.yaml

apiVersion: v1
apiVersion: apps/v1
kind: Deployment
metadata:
  name: server-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      component: server
  template:
    metadata:
      labels:
        component: server
    spec:
      containers:
        - name: server
          image: hongbomiao/hongbomiao-server:latest
          ports:
            - containerPort: 5000
          env:
            - name: NODE_ENV
              value: development

server-cluster-ip-service.yaml

kind: Service
metadata:
  name: server-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    component: server
  ports:
    - port: 5000
      targetPort: 5000

ingress-service.yaml

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: server-cluster-ip-service
                port:
                  number: 5000

I got my minikube IP by

➜ minikube ip
192.168.64.12

When I open 192.168.64.12 in my browser, I got 502 Bad Gateway.

I got some debug idea after reading https://cloud.google.com/kubernetes-engine/docs/how-to/exposing-apps#kubectl-apply. Here is what I have tried:

➜ kubectl get service
NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kubernetes                  ClusterIP   10.96.0.1      <none>        443/TCP    5h34m
server-cluster-ip-service   ClusterIP   10.102.5.161   <none>        5000/TCP   4h39m

➜ kubectl get pods
NAME                                READY   STATUS    RESTARTS   AGE
server-deployment-bc6777445-pj59f   1/1     Running   0          4h39m

➜ kubectl exec -it server-deployment-bc6777445-pj59f -- sh
/app # apk add --no-cache curl
...
/app # curl 10.102.5.161:5000
curl: (28) Failed to connect to 10.102.5.161 port 5000: Operation timed out

It seems my deployment pod has issue connecting ClusterIP service now. Any help will be nice!

1
dockerkubernetesyamlminikube
When you say "it listens at localhost:5000", is that something that's specifically in your application code (maybe a listen() or bind() call) or its configuration? In general if a containerized process listens on a localhost-type address, it won't be reachable from outside its container; you need to set it to listen to 0.0.0.0:5000 instead. You should see the same behavior if you docker run the image without Kubernetes. – David Maze
Did you enable Ingress addon in your minikube cluster? Could you try your deployment with Nginx image or helloworld image? Maybe this thread might help you. – PjoterS
@DavidMaze I added more details. It is a Express.js app using listen at port 5000. I met another issue, will report back once I found the solution for that. – Hongbo Miao
@PjoterS yes, I did enable Ingress addon. However, not there yet. Because my issue is my deployment pod has issue connecting ClusterIP service. – Hongbo Miao
What is the output of kubectl describe ingress ingress-service? why you are using ClusterIP? ClusterIp is an internal service – Lety

1 Answers

2
votes

It turns out the issue is caused by my VPN.

I didn't change anything for the Kubernetes config in my question.

Also, letting the Express.js server explicitly listen at 0.0.0.0 is not necessary neither.

(Note @David Maze's comment under the question about 0.0.0.0 is still valuable)

const app = express()
  .use(bodyParser.json())
  .use(express.static(path.join(__dirname, '../dist')))

app.listen(5000); // This just works. No need explicitly change to app.listen(5000, '0.0.0.0');

At the time of writing, I was in China. To get rid of the VPN while still using Kubenetes / minikube, I found a way and posted it at GitHub here.

After turning off the VPN with this workaround solution, everything just works.

Copy my solution using minikube in China here:

Step 1 - Download the Aliyun version minikube

curl -Lo minikube https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/releases/v1.14.2/minikube-darwin-amd64 && chmod +x minikube && sudo mv minikube /usr/local/bin/

Note: You can find if there is a new version to replace v1.14.2 in the command above at https://github.com/AliyunContainerService/minikube/wiki#%E5%AE%89%E8%A3%85minikube

Step 2 - Start the minikube

minikube start --image-mirror-country cn \
 --driver=hyperkit \
 --iso-url=https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/iso/minikube-v1.15.0.iso \
 --registry-mirror=https://xxxxxxxx.mirror.aliyuncs.com

Note 1: You can find latest minikube version at https://github.com/kubernetes/minikube/blob/master/CHANGELOG.md, then replace v1.15.0 in the command above to newer version.

However, Aliyun's minikube version is a little behind. To verify if a new version exists, you can replace the version in the URL of https://kubernetes.oss-cn-hangzhou.aliyuncs.com/minikube/iso/minikube-v1.15.0.iso to different new versions, such as v1.15.1, and then open it in the browser.

Note 2: For the xxxxxxxx in the command above, you can find yours at https://cr.console.aliyun.com/cn-hangzhou/instances/mirrors (Need register an Aliyun account first)

Note 3: You can pass more parameters to this Aliyun version minikube start, check at https://github.com/AliyunContainerService/minikube/wiki#%E5%90%AF%E5%8A%A8

In my case, I am using the driver hyperkit on macOS, and Aliyun's iso-url, registry-mirror to speed up.