Kubernetes Readiness probe failed: dial tcp 10.244.0.10:5000: connect: connection refused

2
votes

I have a simple service and pod as described below but the readiness probe fails complaining for connection refused

apiVersion: v1
kind: Service
metadata:
  name: keystone-api
spec:
  selector:
    app: keystone
  ports:
    - protocol: TCP
      port: 5000
      targetPort: 5000
      name: public
    - protocol: TCP
      port: 35357
      targetPort: 35357
      name: admin
...
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keystone
  labels:
    app: keystone
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keystone
  template:
    metadata:
      labels:
        app: keystone
    spec:
      containers:
        - name: keystone
          image: openio/openstack-keystone
          readinessProbe:
            tcpSocket:
              port: 5000
          env:
            - name: OS_IDENTITY_ADMIN_PASSWD
              value: password
            - name: IPADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - containerPort: 5000
              name: public
            - containerPort: 35357
              name: admin

Error:

 Normal   Pulled     37m                kubelet, kind-pl  Successfully pulled image "openio/openstack-keystone"
  Normal   Created    37m                kubelet, kind-pl  Created container keystone
  Normal   Started    37m                kubelet, kind-pl  Started container keystone
  Warning  Unhealthy  35m (x8 over 37m)  kubelet, kind-pl  Readiness probe failed: dial tcp 10.244.0.10:5000: connect: connection refused

This is how I launched the deployment and service kubectl apply -f application.yaml --namespace=heat

What am i missing here? Service spec

spec:
  clusterIP: 10.96.162.65
  ports:
  - name: public
    port: 5000
    protocol: TCP
    targetPort: 5000
  - name: admin
    port: 35357
    protocol: TCP
    targetPort: 35357
  selector:
    app: keystone
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

From my VM: telnet 10.96.162.65 5000 Trying 10.96.162.65...

Kubectl describe pod logs:

Namespace:    heat
Priority:     0
Node:         kind-control-plane/172.17.0.2
Start Time:   Sun, 19 Apr 2020 16:04:36 +0530
Labels:       app=keystone
              pod-template-hash=8587f8dc76
Annotations:  <none>
Status:       Running
IP:           10.244.0.10
IPs:
  IP:           10.244.0.10
Controlled By:  ReplicaSet/keystone-8587f8dc76
Containers:
  keystone:
    Container ID:   containerd://9888e62ac7df3f076bd542591a6413a0ef5b70be2c792bbf06e423b5dae89ca0
    Image:          openio/openstack-keystone
    Image ID:       docker.io/openio/openstack-keystone@sha256:62c8e36046ead4289ca4a6a49774bc589e638f46c0921f40703570ccda47a320
    Ports:          5000/TCP, 35357/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Sun, 19 Apr 2020 16:08:01 +0530
    Ready:          True
    Restart Count:  0
    Readiness:      tcp-socket :5000 delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      OS_IDENTITY_ADMIN_PASSWD:  password
      IPADDR:                     (v1:status.podIP)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wf2bp (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-wf2bp:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-wf2bp
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

## Kubectl log podname logs:

10.244.0.10 - - [19/Apr/2020 11:14:33] "POST /v3/auth/tokens HTTP/1.1" 201 2161
2020-04-19 11:14:33.699 49 INFO keystone.common.wsgi [req-fc64c89f-724c-4838-bc34-3907a8f79041 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] GET http://10.244.0.10:35357/v3/services/heat
2020-04-19 11:14:33.705 49 WARNING keystone.common.wsgi [req-fc64c89f-724c-4838-bc34-3907a8f79041 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] Could not find service: heat.: ServiceNotFound: Could not find service: heat.
10.244.0.10 - - [19/Apr/2020 11:14:33] "GET /v3/services/heat HTTP/1.1" 404 90
2020-04-19 11:14:33.970 49 INFO keystone.common.wsgi [req-3589e675-8818-4b82-ad7d-c944d9e2a232 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] GET http://10.244.0.10:35357/v3/services?name=heat
10.244.0.10 - - [19/Apr/2020 11:14:34] "GET /v3/services?name=heat HTTP/1.1" 200 341
2020-04-19 11:14:34.210 49 INFO keystone.common.wsgi [req-492a3e9f-8892-4204-8ca9-c1465e28e709 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] POST http://10.244.0.10:35357/v3/endpoints
10.244.0.10 - - [19/Apr/2020 11:14:34] "POST /v3/endpoints HTTP/1.1" 201 360
10.244.0.10 - - [19/Apr/2020 11:14:38] "GET / HTTP/1.1" 300 267
2020-04-19 11:14:38.089 49 INFO keystone.common.wsgi [req-4c8952b3-7d5b-4ee3-9cf9-f736e1628448 - - - - -] POST http://10.244.0.10:35357/v3/auth/tokens
10.244.0.10 - - [19/Apr/2020 11:14:38] "POST /v3/auth/tokens HTTP/1.1" 201 2367
2020-04-19 11:14:38.737 49 INFO keystone.common.wsgi [req-ebd817f5-d473-4909-b04d-ff0e1d5badab - - - - -] POST http://10.244.0.10:35357/v3/auth/tokens
10.244.0.10 - - [19/Apr/2020 11:14:39] "POST /v3/auth/tokens HTTP/1.1" 201 2367
2020-04-19 11:14:39.635 49 INFO keystone.common.wsgi [req-b68139dc-c62f-4fd7-9cfc-e472a88b9022 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] GET http://10.244.0.10:35357/v3/services/heat
2020-04-19 11:14:39.640 49 WARNING keystone.common.wsgi [req-b68139dc-c62f-4fd7-9cfc-e472a88b9022 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] Could not find service: heat.: ServiceNotFound: Could not find service: heat.
10.244.0.10 - - [19/Apr/2020 11:14:39] "GET /v3/services/heat HTTP/1.1" 404 90
2020-04-19 11:14:39.814 49 INFO keystone.common.wsgi [req-6562f24f-f032-4150-86d9-951318918871 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] GET http://10.244.0.10:35357/v3/services?name=heat
10.244.0.10 - - [19/Apr/2020 11:14:39] "GET /v3/services?name=heat HTTP/1.1" 200 341
2020-04-19 11:14:40.043 49 INFO keystone.common.wsgi [req-6542d767-29bf-4c1a-bbd9-a81a72e106dc 411ecaea9d3241a88e86355ba22f7a0f 277a0fe02d174c47bae4d67e697be0a7 - default default] POST http://10.244.0.10:35357/v3/endpoints
10.244.0.10 - - [19/Apr/2020 11:14:40] "POST /v3/endpoints HTTP/1.1" 201 362
Have manually created heat service 
[root@keystone-8587f8dc76-rthmn /]# openstack service list                                                          
+----------------------------------+--------------+---------------+
| ID                               | Name         | Type          |
+----------------------------------+--------------+---------------+
| ec5ad9402b3b46599f3f8862e79429b3 | keystone     | identity      |
| 625d8b82a67d472981789f10ba37c381 | openio-swift | object-store  |
| 415b33b5d45c48f6916d38f7b146953a | heat         | orchestration |
+----------------------------------+--------------+---------------+
2
kubernetesopenstackkubernetes-helmkubernetes-podkeystone
share output of kubectl logs podname and Kubectl describe pod podname – Arghya Sadhu
Your image is not listening on port 5000! – suren
Updated logs and describe of pod above – Saurabh Arora
@Suren: what do you mean by Image is not listening on port 5000! I can see that in container curl http://10.244.0.10:5000 {"versions": {"values": [{"status": "stable", "updated": "2018-10-15T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.11", "links": [{"href": "http://10.244.0.10:5000/v3/", "rel": "self"}]}]}} – Saurabh Arora
@Saurabh Arora that was not the result with my tests. if there would be anything there, you would never get connection refused. You might get some other error, but not connection refused. – suren

2 Answers

4
votes

TL;DR:

I've made some tests, your docker image and deployment seems really fine ,I was able to log into the pod, it was running and listening on the port.

  • The reason why your readiness probe was returning Warning Unhealthy...: connection refused was because it was not given enough time for the pod to start.

I edited your deployment with the following lines:

          readinessProbe:
            tcpSocket:
              port: 5000
            initialDelaySeconds: 300
            periodSeconds: 30

Explanation:

initialDelaySeconds: Number of seconds after the container has started before liveness or readiness probes are initiated. Defaults to 0 seconds. Minimum value is 0.

periodSeconds: How often (in seconds) to perform the probe. Default to 10s. Minimum value is 1s.

NOTE: During my tests I noticed that the pod takes about 5 minutes to be running, way longer than the default 10s, that's why I set it as 300 seconds.

Meaning that after 5 minutes the pod was serving on port 5000.

Add the initialDelaySeconds line to your deployment and you should be fine.

Here is my Reproduction:

  • Edited Deployment:
apiVersion: v1
kind: Service
metadata:
  name: keystone-api
spec:
  selector:
    app: keystone
  ports:
    - protocol: TCP
      port: 5000
      targetPort: 5000
      name: public
    - protocol: TCP
      port: 35357
      targetPort: 35357
      name: admin
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: keystone
  labels:
    app: keystone
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keystone
  template:
    metadata:
      labels:
        app: keystone
    spec:
      containers:
        - name: keystone
          image: openio/openstack-keystone
          readinessProbe:
            tcpSocket:
              port: 5000
            initialDelaySeconds: 300
            periodSeconds: 30
          env:
            - name: OS_IDENTITY_ADMIN_PASSWD
              value: password
            - name: IPADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - containerPort: 5000
              name: public
            - containerPort: 35357
              name: admin
  • Create the resource and wait:
$ kubectl get pods  -w
NAME                        READY   STATUS    RESTARTS   AGE
keystone-7fd895cfb5-kqnnn   0/1     Running   0          3m28s
ubuntu                      1/1     Running   0          113m
keystone-7fd895cfb5-kqnnn   1/1     Running   0          5m4s
  • After 5min4s the container was running 1/1 and I describe the pod:
$ kubectl describe pod keystone-586b8948d5-c4lpq
Name:         keystone-586b8948d5-c4lpq
Namespace:    default
Priority:     0
Node:         minikube/192.168.39.39
Start Time:   Mon, 20 Apr 2020 15:02:24 +0000
Labels:       app=keystone
              pod-template-hash=586b8948d5
Annotations:  <none>
Status:       Running
IP:           172.17.0.7
IPs:
  IP:           172.17.0.7
Controlled By:  ReplicaSet/keystone-586b8948d5
Containers:
  keystone:
    Container ID:   docker://8bc14d2b6868df6852967c4a68c997371006a5d83555c500d86060e48c549165
    Image:          openio/openstack-keystone
    Image ID:       docker-pullable://openio/openstack-keystone@sha256:62c8e36046ead4289ca4a6a49774bc589e638f46c0921f40703570ccda47a320
    Ports:          5000/TCP, 35357/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Mon, 20 Apr 2020 15:02:26 +0000
    Ready:          True
    Restart Count:  0
    Readiness:      tcp-socket :5000 delay=300s timeout=1s period=30s #success=1 #failure=3
    Environment:
      OS_IDENTITY_ADMIN_PASSWD:  password
      IPADDR:                     (v1:status.podIP)
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-kcw8c (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-kcw8c:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-kcw8c
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age        From               Message
  ----    ------     ----       ----               -------
  Normal  Scheduled  <unknown>  default-scheduler  Successfully assigned default/keystone-586b8948d5-c4lpq to minikube
  Normal  Pulling    7m12s      kubelet, minikube  Pulling image "openio/openstack-keystone"
  Normal  Pulled     7m11s      kubelet, minikube  Successfully pulled image "openio/openstack-keystone"
  Normal  Created    7m11s      kubelet, minikube  Created container keystone
  Normal  Started    7m11s      kubelet, minikube  Started container keystone

As you can see now there is no error.

Let me know in the comments if you have any doubt.

0
votes

I checked with the docker instructions, and it did run, but somehow the app was not getting deployed correctly. To check with docker it is straight forward because they are using host network, so from you host you can do netstat, and you will see that there is nothing listening on port 5000.

I accessed the container and ran the init script (keystone-v3.sh) again, and it started to work. I did the same on kubernetes Deployment, and it worked too.

So, this is your functional Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: keystone
  labels:
    app: keystone
spec:
  replicas: 1
  selector:
    matchLabels:
      app: keystone
  template:
    metadata:
      labels:
        app: keystone
    spec:
      containers:
        - name: keystone
          image: openio/openstack-keystone
          command: ["./keystone-v3.sh"]     #<- you add this line
          readinessProbe:
            tcpSocket:
              port: 5000
          env:
            - name: OS_IDENTITY_ADMIN_PASSWD
              value: password
            - name: IPADDR
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          ports:
            - containerPort: 5000
              name: public
            - containerPort: 35357
              name: admin

~$ kubectl get po
NAME                        READY   STATUS    RESTARTS   AGE
alpine-786c6d498d-dsxfh     1/1     Running   1          11d
curler-755cc7cfff-fwz4g     1/1     Running   1          11d
keystone-6d997f4f8c-5kkxc   1/1     Running   0          26m
nginx-6db489d4b7-jlhql      1/1     Running   1          11d
~$ kubectl logs --tail 5 keystone-6d997f4f8c-5kkxc
********************************************************************************
STARTING test server keystone.server.wsgi.initialize_public_application
Available at http://keystone-6d997f4f8c-5kkxc:5000/
DANGER! For testing only, do not use in production
********************************************************************************
~$

Or you try to fix it from the image, but I guess that's not your repo right?

UPDATE

Check on this:

~$ sudo docker run -d --net=host -e IPADDR=192.168.56.102 openio/openstack-keystone
Unable to find image 'openio/openstack-keystone:latest' locally
latest: Pulling from openio/openstack-keystone
ab5ef0e58194: Pull complete 
ca37595f2b63: Pull complete 
878ef80688be: Pull complete 
Digest: sha256:62c8e36046ead4289ca4a6a49774bc589e638f46c0921f40703570ccda47a320
Status: Downloaded newer image for openio/openstack-keystone:latest
703a05b8fdc8b7294895122b6f369a4d0a6b4582104ed360d6be68d012ea5b3c
~$ netstat -tlpn | grep 5000
NOTE: NOTHING LISTENING ON PORT 5000
~$ sudo docker ps | grep openio
703a05b8fdc8        openio/openstack-keystone   "/keystone-v3.sh"        34 seconds ago      Up 32 seconds                           quizzical_swartz
~$ sudo docker exec -it 703a05b8fdc8 bash
[root@v1-17-master /]# ls   
anaconda-post.log  bin  dev  etc  home  keystone-v3.log  keystone-v3.sh  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
[root@v1-17-master /]# tail keystone-v3.sh 
openstack endpoint create --region "$OS_OBJECTSTORE_SERVICE_REGION" 'object-store' internal "$OS_OBJECTSTORE_URL_INTERNAL"
openstack endpoint create --region "$OS_OBJECTSTORE_SERVICE_REGION" 'object-store' admin    "$OS_OBJECTSTORE_URL_ADMIN"
# Demo user
openstack domain create "$OS_USER_DEMO_DOMAIN"
openstack project create "$OS_USER_DEMO_PROJECT"
openstack user create --password "$OS_USER_DEMO_PASSWD" --project "$OS_USER_DEMO_PROJECT" "$OS_USER_DEMO_USERNAME"
openstack role add --user "$OS_USER_DEMO_USERNAME" --project "$OS_USER_DEMO_PROJECT" "$OS_USER_DEMO_ROLE"

echo '> Starting Keystone public service ...'
/usr/bin/keystone-wsgi-public --port 5000
[root@v1-17-master /]# /usr/bin/keystone-wsgi-public --port 5000 &
[1] 172
[root@v1-17-master /]# exit
~$ sudo netstat -tlpn | grep 5000
tcp        0      0 0.0.0.0:5000            0.0.0.0:*               LISTEN      10207/python2