Kubernetes - connection refused diagnosis

Question

I have a running GCP Kubernetes cluster. I managed to deploy some services and exposed them successfully using kubectl expose ... type="LoadBalancer"... However, one particular new service is not working. I know there may be a thousand causes to check, but the Docker images I build are very compact so I can't find useful tools to run via kubectl exec inside a pod or container.

Question: what might be my diagnosis options using any possible cluster tool only? What kind of logs can I inspect or which environmental variables can I read?

UPDATED:

$ kubectl get pods

NAME                               READY     STATUS    RESTARTS   AGE
helianto-mailer-1024769093-6407d   2/2       Running   0          6d
helianto-spring-2246525676-l54p9   2/2       Running   0          6d
iservport-shipfo-12873703-wrh37    2/2       Running   0          13h

$ kubectl describe pod iservport-shipfo-12873703-wrh37

Name:           iservport-shipfo-12873703-wrh37
Namespace:      default
Node:           gke-iservport01-default-pool-xxx/xx.xx.xx.xx
Start Time:     Tue, 14 Mar 2017 17:28:18 -0300
Labels:         app=SHIPFO
                pod-template-hash=12873703
Status:         Running
IP:             yy.yy.yy.yy
Controllers:    ReplicaSet/iservport-shipfo-12873703
Containers:
  iservport-shipfo:
    Container ID:           docker://...
Image:              us.gcr.io/mvps-156214/iservport-xxx
Image ID:           docker://...
    Port:               8085/TCP
    Requests:
      cpu:              100m
    State:              Running
      Started:          Tue, 14 Mar 2017 17:28:33 -0300
    Ready:              True
    Restart Count:      0
    Volume Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mmeza (ro)
    Environment Variables:
  SPRING_PROFILES_ACTIVE:   gcp
      HELIANTO_MAILER_URL:      http://10.35.254.197:8082
  cloudsql-proxy:
    Container ID:       docker://...
    Image:              b.gcr.io/cloudsql-docker/gce-proxy:1.05
    Image ID:           docker://...
    Port:               
    Command:
  /cloud_sql_proxy
  --dir=/cloudsql
  -instances=mvps-156214:us-east1-b:helianto01=tcp:3306
  -credential_file=/secrets/cloudsql/credentials.json
    Requests:
      cpu:              100m
    State:              Running
      Started:          Tue, 14 Mar 2017 17:28:33 -0300
    Ready:              True
    Restart Count:      0
    Volume Mounts:
      /cloudsql from cloudsql (rw)
      /etc/ssl/certs from ssl-certs (rw)
      /secrets/cloudsql from cloudsql-oauth-credentials (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-mmeza (ro)
    Environment Variables:      <none>
Conditions:
Type          Status
  Initialized   True 
  Ready         True 
  PodScheduled  True 
Volumes:
  cloudsql-oauth-credentials:
    Type:       Secret (a volume populated by a Secret)
    SecretName: cloudsql-oauth-credentials
  ssl-certs:
    Type:       HostPath (bare host directory volume)
    Path:       /etc/ssl/certs
  cloudsql:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
  default-token-mmeza:
    Type:       Secret (a volume populated by a Secret)
    SecretName: default-token-mmeza
QoS Class:      Burstable
Tolerations:    <none>
No events.

$ kubectl get svc

NAME                      CLUSTER-IP      EXTERNAL-IP       PORT(S)                      AGE
helianto-mailer-service   10.35.254.197   <nodes>           443:32178/TCP,80:30771/TCP   12d
helianto-spring           10.35.241.27    xxx.xxx.xxx.xxx   80:30974/TCP                 52d
iservport-shipfo          10.35.240.129   xx.xxx.xxx.xxx    80:32598/TCP                 14h
kubernetes                10.35.240.1     <none>            443/TCP                      53d

$ kubectl describe svc iservport-shipfo

Name:                   iservport-shipfo
Namespace:              default
Labels:                 app=SHIPFO
Selector:               app=SHIPFO
Type:                   LoadBalancer
IP:                     10.35.240.129
LoadBalancer Ingress:   xx.xxx.xxx.xxx
Port:                   <unset> 80/TCP
NodePort:               <unset> 32598/TCP
Endpoints:              10.32.4.26:8085
Session Affinity:       None
No events.

Geovana Souza Geovana Souza · Accepted Answer · 2017-03-15T12:16:33

You need make sure if your service is responding in http port. Maybe you could do a port-forward from your pod to your desktop local. Please, replace the values pod_name, pod_port and local_port in command bellow.

kubectl port-forward <pod_name> <local_port>:<pod_port>

After this, access http://localhost:local_port and verify if return something. This way, you can make sure if your application is responding.

Kubernetes - connection refused diagnosis

2 Answers