Rewrites with path regex are not working in EKS and nginx ingress controller

Question

I am using EKS cluster with kubernetes version 1.18 and nginx igress controller. I have created below delpoyment, service and ingress resource what I have noticed ingress resource is not working for paths with regex and it is always returning 404 from ingress controller itself without passing request to backend pod. I have verified using describe I am able to see proper backends.

Can you please help on this.

deployment.yaml

apiVersion: apps/v1
   kind: Deployment
   metadata:
      name: "wildfly"
      namespace: "sit-web-n"
   spec:
      selector:
        matchLabels:
        app: "wildfly"
      replicas: 1
      template:
        metadata:
      labels:
        app: "wildfly"
     spec:
      containers:
      - image: <account>.dkr.ecr.eu-west-1.amazonaws.com/ecr-test:wildfly
        imagePullPolicy: Always
        name: "wildfly"
        env:
        - name: JAVA_OPTS
          value: "-Xms128m -Xmx768m"
        ports:
        - containerPort: 8080

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: "wildfly"
  namespace: "sit-web-n"
spec:
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP
  type: NodePort
  selector:
    app: "wildfly"

ingress.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: wildfly
  namespace: "sit-web-n"
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  ingressClassName: nginx
  tls:
    - hosts:
      - wildfly.eks-test.co.uk
      secretName: tls-secret
   rules:
   - host: wildfly.eks-test.co.uk
    http:
      paths:
      - path: /customer-svc/?(.*)
        pathType: Prefix
        backend:
          serviceName: wildfly
          servicePort: 8080

Service with backend

kubectl describe svc wildfly  -n sit-web-n
Name:                     wildfly
Namespace:                sit-web-n
Labels:                   <none>
Annotations:              Selector:  app=wildfly
Type:                     NodePort
IP:                       10.100.78.242
Port:                     <unset>  8080/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  31350/TCP
Endpoints:                192.168.0.89:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>
[test]#

ingress

 kubectl describe ingress wildfly  -n sit-web-n
   Name:             wildfly
   Namespace:        sit-web-n
   Address:
   Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
    TLS:
     tls-secret terminates wildfly.eks-test.co.uk
    Rules:
     Host                    Path  Backends
      ----                    ----  --------
     wildfly.eks-test.co.uk
                            /customer-svc/?(.*)   wildfly:8080 (192.168.0.89:8080)
   Annotations:              nginx.ingress.kubernetes.io/force-ssl-redirect: false
                             nginx.ingress.kubernetes.io/rewrite-target: /$1
                             nginx.ingress.kubernetes.io/ssl-redirect: false
    Events:
     Type    Reason          Age   From                      Message
      ----    ------          ----  ----                      -------
      Normal  AddedOrUpdated  14m   nginx-ingress-controller  Configuration for sit-web-n/wildfly was added or updated

curl output

curl -ik -H 'Host: wildfly.eks-test.co.uk' https://ip-192-168-0-101.eu-west-1.compute.internal/custom- 
   svc/health
    HTTP/1.1 404 Not Found
    Server: nginx
    Date: Tue, 17 Nov 2020 08:31:08 GMT
   Content-Type: text/html
   Content-Length: 146
   Connection: keep-alive
   Strict-Transport-Security: max-age=15724800; includeSubDomains; preload

   <html>
   <head><title>404 Not Found</title></head>
   <body>
   <center><h1>404 Not Found</h1></center>
   <hr><center>nginx</center>
   </body>
   </html>



nginx ingress controller logs -

2020/11/17 08:49:44 [error] 52#52: *17 open() "/etc/nginx/html/custom-svc/health" failed (2: No such file or directory), client: 172.31.0.183, server: wildfly.eks-test.co.uk, request: "GET /custom-svc/health HTTP/1.1", host: "wildfly.eks-test.co.uk"
wildfly.eks-test.co.uk 172.31.0.183 - - [17/Nov/2020:08:49:44 +0000] GET /custom-svc/health HTTP/1.1 404 146 0.000 - curl/7.61.1 - - -
2020/11/17 08:49:48 [error] 52#52: *18 open() "/etc/nginx/html/custom-svc/health" failed (2: No such file or directory), client: 172.31.0.183, server: wildfly.eks-test.co.uk, request: "GET /custom-svc/health HTTP/1.1", host: "wildfly.eks-test.co.uk"
wildfly.eks-test.co.uk 172.31.0.183 - - [17/Nov/2020:08:49:48 +0000] GET /custom-svc/health HTTP/1.1 404 146 0.000 - curl/7.61.1 - - -

1. What is your controller version. 2. Those logs looks like not from nginx ingress controller but from your backend. 3. You are using this path "custom-svc/health" in your curl command instead of path: /customer-svc/. 4 Please verify your path directly on the backend using curl like curl localhost:8080/customer-svc/health vs curl localhost:8080/custom-svc/health using rewrite-target it should be even curl localhost:8080/health — Mark

Mark Mark · Accepted Answer · 2020-11-20T14:58:57

As stated in the comment:

1. Inside ingress resource you have defined path: customer-svc while trying to reach custom-svc

2. "/etc/nginx/html/custom-svc/health" failed (2: No such file or directory)". Normally such errors inform us that there is no such path on the underlying backend (strange for nginx ingress controller).

It's hard to figure all possibilities but try to follow this:

Make sure you have installed and configured proper nginx controller. Different controllers and versions have different configurations and options like:

attention Starting in Version 0.22.0, ingress definitions using the annotation nginx.ingress.kubernetes.io/rewrite-target are not backwards compatible with previous versions. In Version 0.22.0 and beyond, any substrings within the request URI that need to be passed to the rewritten path must explicitly be defined in a capture group.

In addition please take a look at:

ingressClassName configuration,
Default Ingress Class configuration,

You can mark a particular IngressClass as default for your cluster. Setting the ingressclass.kubernetes.io/is-default-class annotation to true on an IngressClass resource will ensure that new Ingresses without an ingressClassName field specified will be assigned this default IngressClass.

Update:

As stated above: "Different controllers and versions have different configurations and options". You are using nginxinc kubernetes-ingress controller which is different from the NGINX Ingress controller in kubernetes/ingress-nginx repo. Please take a look at this doc to find out about the key differences.

Those annotations:

nginx.ingress.kubernetes.io/ssl-redirect: "false"
nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
nginx.ingress.kubernetes.io/rewrite-target: /$1

were taken from nginx-ingress (kubernetes) and thus will not work with nginxinc kubernetes-ingress controller.

As an example for uri rewriting please use nginx.org/rewrites annotation:

annotations:
  nginx.org/rewrites: "serviceName=wildfly rewrite=/"

In addition, for using nginxinc kubernetes-ingress controller with regex and advanced routing support please take a look at VirtualServer.Route custom resource definition (CRD) concept. This functionality requires to install/configure CRD support during controller installation.

Here you can find complete guide how it should be configured using VirtualServer CRD for nginxinc kubernetes-ingress controller.

Additional information:

controllers

Tutorials: