Nginx Controller upgrade is failing on Kubernetes 1.16.10 cluster

Question

I'm trying to upgrade the Nginx controller on a Kubernetes cluster version v1.16 (v1.16.10) and unfortunately it was not succeeded.

My Nginx setup is configured as a DaemonSet with the helm stable repository, since the new repository as changed to https://kubernetes.github.io/ingress-nginx I'm trying to use the new repo and trying to upgrade the version to at lease 0.33 which is helm release 2.10.0

Error behavior:

upgrade is half way succeeded and stuck in a place where saying "Pod is not ready: kube-system/nginx-ingress-controller-xxxx" in the helm controller. At that time there were pods created by the DaemonSet in the nodes and they were going to the "CrashLoopBackOff" state and then "Error" state, logs displayed the below error:

W0928 05:21:50.497500       6 flags.go:249] SSL certificate chain completion is disabled (--enable-ssl-chain-completion=false)
W0928 05:21:50.497572       6 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0928 05:21:50.497777       6 main.go:218] Creating API client for https://172.31.0.1:443
I0928 05:21:50.505692       6 main.go:262] Running in Kubernetes cluster version v1.16 (v1.16.10) - git (clean) commit f3add640dbcd4f3c33a7749f38baaac0b3fe810d - platform linux/amd64
I0928 05:21:50.512138       6 main.go:85] Validated kube-system/nginx-ingress-ingress-nginx-defaultbackend as the default backend.
F0928 05:21:50.517958       6 main.go:91] No service with name kube-system found in namespace nginx-ingress-ingress-nginx-controller: services "nginx-ingress-ingress-nginx-controller" not found

I can confirm that there were no service running for the Nginx Controller with my current helm release (1.33.1). I'm not sure this service is an essential aspect for this version with the NodePort configuration or I'm missing something here.

Service is set to false in my current config in the DaemonSet

      service:
        enabled: false

I found that there were few issues in k8s 1.16 and not sure this is also something related to that. Also, I can confirm that the default backend is registerig successfully as few of the issues I found related to that as well and it couldn't be the reason.

Really appreacite your kind and helpful thoughts here. Thanks.

I guess its your local environment. Are you using Kubeadm or Minikube? You've used 0.22 Nginx and wanted to upgrade to 0.33? How did you do that? Did you try to remove old Helm chart and deploy new one? Exact steps you did would be helpful. — PjoterS
No it's not Minikube it Kubeadm and it is with multiple Master and worker nodes deployed with the HA cluster. I use Helm Controller and changed the repo and the version to new one. That's how I did it. Do you know any changes to the Service in those Helm releases. May be adding a requirement of a service? or to avoid creating a service any option to set in the values? Sorry if I made you confused happy to clear any doubts. — Aruna Lakmal
even though, if I run this manually in a cluster with this command using helm3 getting the same issue. helm install ingress ingress-nginx/ingress-nginx --version 2.10.0 --set controller.service.enabled=false --set controller.kind=DaemonSet — Aruna Lakmal
Old configuration using the stable repo working perfectly with this: helm install old-ingress stable/nginx-ingress --version 1.33.1 --set controller.service.enabled=false --set controller.kind=DaemonSet deployed the pods without a service. Any thoughts? I'm stuck here. — Aruna Lakmal

Aruna Lakmal Aruna Lakmal · Accepted Answer · 2020-09-30T11:45:01

Finally, I was able to figure it out and working perfectly. It was due to the change from chart version 0.32 to 0.33 and it is checking the publish-service flag is defined.

--set controller.publishService.enabled=false

The above parameter should be explicitly set to avoid this.

Related PR: #5553

working command:

helm install nginx-new ingress-nginx/ingress-nginx --version 3.3.0 --set controller.service.enabled=false --set controller.kind=DaemonSet --set controller.publishService.enabled=false

Nginx Controller upgrade is failing on Kubernetes 1.16.10 cluster

1 Answers