Unable to access Kubernetes ClusterIP services via nginx-ingress-controller

Question

I'm a Kubernetes amateur trying to use NGINX ingress controller on GKE. I'm following this google cloud documentation to setup NGINX Ingress for my services, but, I'm facing issues in accessing the NGINX locations.

What's working?

Ingress-Controller deployment using Helm (RBAC enabled)
ClusterIP service deployments

What's not working?

Ingress resource to expose multiple ClusterIP services using unique paths (fanout routing)

K8S Services

[msekar@ebs kube-base]$ kubectl get services -n payment-gateway-7682352
NAME                            TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
nginx-ingress-controller        LoadBalancer   10.35.241.255   35.188.161.171   80:31918/TCP,443:31360/TCP   6h
nginx-ingress-default-backend   ClusterIP      10.35.251.5     <none>           80/TCP                       6h
payment-gateway-dev             ClusterIP      10.35.254.167   <none>           5000/TCP                     6h
payment-gateway-qa              ClusterIP      10.35.253.94    <none>           5000/TCP                     6h

K8S Ingress

[msekar@ebs kube-base]$ kubectl get ing -n payment-gateway-7682352
NAME                HOSTS     ADDRESS          PORTS     AGE
pgw-nginx-ingress   *         104.198.78.169   80        6h

[msekar@ebs kube-base]$ kubectl describe ing pgw-nginx-ingress -n payment-gateway-7682352
Name:             pgw-nginx-ingress
Namespace:        payment-gateway-7682352
Address:          104.198.78.169
Default backend:  default-http-backend:80 (10.32.1.4:8080)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     
        /dev/   payment-gateway-dev:5000 (<none>)
        /qa/    payment-gateway-qa:5000 (<none>)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/ssl-redirect":"false"},"name":"pgw-nginx-ingress","namespace":"payment-gateway-7682352"},"spec":{"rules":[{"http":{"paths":[{"backend":{"serviceName":"payment-gateway-dev","servicePort":5000},"path":"/dev/"},{"backend":{"serviceName":"payment-gateway-qa","servicePort":5000},"path":"/qa/"}]}}]}}

  kubernetes.io/ingress.class:               nginx
  nginx.ingress.kubernetes.io/ssl-redirect:  false
Events:                                      <none>

Last applied configuration in the annotations (ingress description output) shows the ingress resource manifest. But, I'm pasting it below for reference

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: pgw-nginx-ingress
  namespace: payment-gateway-7682352
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: payment-gateway-dev
          servicePort: 5000
        path: /dev/
      - backend:
          serviceName: payment-gateway-qa
          servicePort: 5000
        path: /qa/

Additional Info

The services I'm trying to access are springboot services that use contexts, so, the root location isn't a valid end-point.

The container's readiness and liveliness probes are defined accordingly.

For example, "payment-gateway-dev" service is using a context /pgw/v1 context, so, the deployment can only be accessed through the context. To access application's swagger spec you would use the URL

http://<>/pgw/v1/swagger-ui.html

Behaviour of my deployment

ingress-controller-LB-ip = 35.188.161.171

Accessing ingress controller load balancer "http://35.188.161.171" takes me to default 404 backend
Accessing ingress controller load balancer health "http://35.188.161.171/healthz" returns 200 HTTP response as expected
Trying to access the services using the URLs below returns "404: page not found" error
- http://35.188.161.171/dev/pgw/v1/swagger-ui.html
- http://35.188.161.171/qa/pgw/v1/swagger-ui.html

Any suggestions about or insights into what I might be doing wrong will be much appreciated.

Fei Fei · Accepted Answer · 2019-03-16T04:32:54

+1 for this well asked question.

Your setup seemed right to me. In you explanation, I could find that your services would require http://<>/pgw/v1/swagger-ui.html as context. However, in your setup the path submitted to the service will be http://<>/qa/pgw/v1/swagger-ui.html if your route is /qa/.

To remove the prefix, what you would need to do is to add a rewrite rule to your ingress:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: pgw-nginx-ingress
  namespace: payment-gateway-7682352
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
  - http:
      paths:
      - backend:
          serviceName: payment-gateway-dev
          servicePort: 5000
        path: /dev/(.+)
      - backend:
          serviceName: payment-gateway-qa
          servicePort: 5000
        path: /qa/(.+)

After this, you service should receive the correct contexts.

Ref:

Rewrite: https://github.com/kubernetes/ingress-nginx/blob/master/docs/examples/rewrite/README.md
Ingress Route Matching: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/ingress-path-matching.md

Unable to access Kubernetes ClusterIP services via nginx-ingress-controller

2 Answers