Migrate docker-compose nginx.conf to kubernetes nginx ingress

Question

I am trying to convert my docker-compose nginx.conf to kubernetes nginx ingress controller. This is my nginx.conf that i am using with docker-compose.

server {
    listen 3000;
    root /srv/www/static;
    sendfile on;
    gzip on;
    gzip_comp_level 1;
    gzip_proxied any;
    gzip_vary on;
    gzip_types application/json text/plain;

    location /api/main {
        rewrite /api/main(.*)$1 break;
        proxy_pass http://main:4000;
        proxy_set_header content-type "application/json";
    }
    location /api/login {
        rewrite /api/login(.*)$1 break;
        proxy_pass http://login:4000;
        proxy_set_header content-type "application/json";
    }
}

This is the nginx ingress controller. I have the ingress addon enabled.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
  - host: test.com
    http:
      paths:
      - path: /api/login(.*)
        pathType: Prefix
        backend:
          service:
            name: login-app
            port:
              number: 4000
      - path: /api/main(.*)
        pathType: Prefix
        backend:
          service:
            name: main-app
            port:
              number: 4000

kubectl get ingress:

NAME             CLASS    HOSTS        ADDRESS   PORTS   AGE
test-ingress   <none>   test.com                 80      20h

kubectl describe ingress test-ingress:

Name:             test-ingress
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  test.com  
              /api/login(.*)   login-app:4000 (10.1.146.185:4000)
              /api/main(.*)    main-app:4000 (10.1.146.186:4000)
Annotations:  nginx.ingress.kubernetes.io/rewrite-target: /$1
Events:       <none>

If I go to test.com:80 I get an nginx 404 error so I believe its up and running but any request that I make to an endpoint also returns 404. I also defined test.com in my local machine etc/hostname and I am also using microk8s --classic instead of minikube.

The way I have it set up is I need to be able to send a request to an endpoint and have url path forwarded to the corresponding service. so test.com/api/main/orders/orderStatus would get rewritten to main-service/api/main/orders/orderStatus but this returns a 404 error I think it might be the way I have the rewrite annotations for kubernetes set up because my nginx.conf for docker-compose works fine.

kubectl -n ingress logs nginx-ingress-microk8s-controller-5h5qm:

NGINX Ingress controller
  Release:       v0.35.0
  Build:         54ad65e32bcab32791ab18531a838d1c0f0811ef
  Repository:    https://github.com/kubernetes/ingress-nginx
  nginx version: nginx/1.19.2

-------------------------------------------------------------------------------

I0210 20:46:20.300962       6 flags.go:205] Watching for Ingress class: public
W0210 20:46:20.301239       6 flags.go:208] Only Ingresses with class "public" will be processed by this Ingress controller
W0210 20:46:20.302300       6 flags.go:252] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0210 20:46:20.302548       6 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0210 20:46:20.431594       6 main.go:231] Creating API client for https://10.152.183.1:443
I0210 20:46:20.492885       6 main.go:275] Running in Kubernetes cluster version v1.20+ (v1.20.1-34+e7db93d188d0d1) - git (clean) commit e7db93d188d0d12f2fe5336d1b85cdb94cb909d3 - platform linux/amd64
I0210 20:46:20.917856       6 main.go:105] SSL fake certificate created /etc/ingress-controller/ssl/default-fake-certificate.pem
I0210 20:46:21.002278       6 main.go:113] Enabling new Ingress features available since Kubernetes v1.18
I0210 20:46:22.130699       6 nginx.go:263] Starting NGINX Ingress controller
I0210 20:46:22.315064       6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress", Name:"nginx-ingress-udp-microk8s-conf", UID:"73215902-b0b2-4da2-b402-5aa68d4a5be6", APIVersion:"v1", ResourceVersion:"668", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress/nginx-ingress-udp-microk8s-conf
I0210 20:46:22.315231       6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress", Name:"nginx-load-balancer-microk8s-conf", UID:"8f8cd494-7d19-4168-8f05-e4992ea3f4de", APIVersion:"v1", ResourceVersion:"666", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress/nginx-load-balancer-microk8s-conf
I0210 20:46:22.401763       6 event.go:278] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress", Name:"nginx-ingress-tcp-microk8s-conf", UID:"885189d0-e5ca-47e2-98f4-8982d813606a", APIVersion:"v1", ResourceVersion:"667", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress/nginx-ingress-tcp-microk8s-conf
I0210 20:46:23.347072       6 store.go:363] ignoring add for ingress test-ingress based on annotation kubernetes.io/ingress.class with value 
I0210 20:46:23.431950       6 nginx.go:307] Starting NGINX process
I0210 20:46:23.432052       6 leaderelection.go:242] attempting to acquire leader lease  ingress/ingress-controller-leader-public...
I0210 20:46:23.444902       6 controller.go:141] Configuration changes detected, backend reload required.
I0210 20:46:25.899037       6 leaderelection.go:252] successfully acquired lease ingress/ingress-controller-leader-public
I0210 20:46:25.899076       6 status.go:86] new leader elected: nginx-ingress-microk8s-controller-5h5qm
I0210 20:46:26.061674       6 controller.go:157] Backend successfully reloaded.
I0210 20:46:26.061807       6 controller.go:166] Initial sync, sleeping for 1 second.
W0210 20:46:27.063786       6 controller.go:184] Dynamic reconfiguration failed: Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
E0210 20:46:27.063892       6 controller.go:188] Unexpected failure reconfiguring NGINX:
Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
W0210 20:46:27.063941       6 queue.go:130] requeuing initial-sync, err Post "http://127.0.0.1:10246/configuration/backends": dial tcp 127.0.0.1:10246: connect: connection refused
I0210 20:46:27.064158       6 controller.go:141] Configuration changes detected, backend reload required.
I0210 20:46:27.398413       6 controller.go:157] Backend successfully reloaded.
I0210 20:46:27.398500       6 controller.go:166] Initial sync, sleeping for 1 second.

Deployment example mainDep.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: main-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: main-app
  template:
    metadata:
      labels:
        app: main-app
    spec:
      containers:
        - name: main-app
          image: peking-main:latest
          imagePullPolicy: Never
          ports:
          - containerPort: 4000
          resources:
            limits:
              memory: "100Mi"
              cpu: "100m"

Service example mainService.yaml

apiVersion: v1
kind: Service
metadata:
  name: main-app
  labels:
    app: main-app
spec:
  type: LoadBalancer
  selector:
    app: main-app
  ports:
    - port: 80
      targetPort: 4000

kubectl get svc

login-app    LoadBalancer   10.152.183.107   <pending>     80:30412/TCP   16m
main-app     LoadBalancer   10.152.183.108   <pending>     80:31369/TCP   16m

kubectl describe svc main-app

Name:                     main-app
Namespace:                default
Labels:                   app=main-app
Annotations:              <none>
Selector:                 app=main-app
Type:                     LoadBalancer
IP Families:              <none>
IP:                       10.152.183.108
IPs:                      10.152.183.108
Port:                     <unset>  80/TCP
TargetPort:               4000/TCP
NodePort:                 <unset>  31369/TCP
Endpoints:                10.1.146.171:4000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Is there an IngressClass defined in the cluster? Not sure it is used on microk8s but I would check if there's one defined or not (kubernetes.io/docs/concepts/services-networking/ingress/…) — AndD
When exactly does it return 404? Is it returned when requesting /api/main ? Can you run kubectl describe ingress ? Is it pointing to correct services? Are pods up and running? — Mariusz K.
@AndD how do I add the ingress class? The example is confusing. — corbin-k
@MariuszK. All the pods and services are up and running. I added more info regarding describing the ingress and routing. — corbin-k
Usually you can set it up by adding an annotation or ingressClassName to the Ingress. But checking microk8s docs I saw that the add on declare a default IngressClass resource, so you don't need to manually specify it, it's automatically used (but you can still search that resource if you want to be sure). Can you post the describe of the services? Is there any nginx-controller Pod you can check the log on? — AndD

corbin-k corbin-k · Accepted Answer · 2021-02-16T01:25:02

So I found a workaround. I changed every port I could find such as application and dockerfile to 80 and I changed the api version of the ingress to v1beta.

‘’’ apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: nginx spec: rules:

host: test.com http: paths:
- path: /api/login backend: serviceName: login-app servicePort: 80
- path: /api/main backend: serviceName: main-app servicePort: 80 ‘’’ Although it works this will throw a warning whenever I apply it.

Migrate docker-compose nginx.conf to kubernetes nginx ingress

1 Answers