Kubernetes NGINX ingress rewrite-target annotation breaking

Question

I am running kubernetes v1.16 under docker desktop for windows. I have installed the nginx-ingress controller v1.7.9 using helm. I have update my hosts file to have the following entry

127.0.0.1 application.local

I have a backend service named hedgehog-service.

The following ingress definition correctly forwards request to the backend.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ml-zoo-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: application.local
    http:
      paths:
      - path: /hedgehog/
        backend:
          serviceName: hedgehog-service
          servicePort: 80

curl application.local/hedgehog works as expected and hits the backend service.

However in order to correctly use the backend service I need to rewrite the target removing the url prefix /hedgehog. Hence I have the following ingress definition

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ml-zoo-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$2
spec:
  rules:
  - host: application.local
    http:
      paths:
      - path: /hedgehog(/|$)(.*)
        backend:
          serviceName: hedgehog-service
          servicePort: 80

As indicated here: https://kubernetes.github.io/ingress-nginx/examples/rewrite/#rewrite-target

Now when I curl application.local/hedgehog/test the ingress controller does not communicate with the backend but according to the logs attempts to open a file

2020/06/23 12:46:48 [error] 708#708: *792 open() "/etc/nginx/html/hedgehog/test" failed (2: No such file or directory), client: 192.168.65.3, server: application.local, request: "GET /hedgehog/test HTTP/1.1", host: "application.local"
192.168.65.3 - - [23/Jun/2020:12:46:48 +0000] "GET /hedgehog/test HTTP/1.1" 404 153 "-" "curl/7.65.3" "-"

Here is the content of etc/nginx/conf.d/default-ml-zoo-ingress

# configuration for default/ml-zoo-ingress

upstream default-ml-zoo-ingress-application.local-hedgehog-service-80 {
        zone default-ml-zoo-ingress-application.local-hedgehog-service-80 256k;
        random two least_conn;

        server 10.1.0.48:80 max_fails=1 fail_timeout=10s max_conns=0;
}

server {
        listen 80;
        server_tokens on;
        server_name application.local;

        location /hedgehog(/|$)(.*) {

                proxy_http_version 1.1;
                proxy_connect_timeout 60s;
                proxy_read_timeout 60s;
                proxy_send_timeout 60s;
                client_max_body_size 1m;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Host $host;
                proxy_set_header X-Forwarded-Port $server_port;
                proxy_set_header X-Forwarded-Proto $scheme;
                proxy_buffering on;

                proxy_pass http://default-ml-zoo-ingress-application.local-hedgehog-service-80;

        }
}

Does anyone know why my URLs are not getting rewritten and the requsts delivered to the backend service?

Thanks in advance!

Joe Joe · Accepted Answer · 2020-06-23T13:13:55

OK having played around with this for hours, once I had written the question my next google turned up an answer.

I installed nginx using helm with the following repo stable/nginx-ingress. However according to this issue https://github.com/kubernetes/ingress-nginx/issues/5756 that is in fact a legacy repository. I uninstalled my controller and changed the repository to ingress-nginx

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

and everything appears to be working as expected. Still not sure why the previous controller installation failed but I can get back to work :)

EDIT: For the aid of others that might end up here - in hindsight I am wondering if the reinstallation simply meant that I deleted and recreated my ingress which might have solved the original problem. In other words make sure you try recreating the ingress before reinstalling the ingress controller with helm.

Kubernetes NGINX ingress rewrite-target annotation breaking

1 Answers