Kubernetes Ingress Controller: Failed calling webhook, dial tcp connect: connection refused

Question

I have set up a Kubernetes cluster (a master and a worker) on two Centos 7 machines. They have the following IPs: Master: 192.168.1.40 Worker: 192.168.1.41

They are accessible by SSH and I am not using a VPN. For both boxes, I have sudo access.

For the work I am doing, I had to add an Nginx Ingress Controller, which I did by doing:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.43.0/deploy/static/provider/baremetal/deploy.yaml

This yaml file seems fine to me and is a common one that occurs when trying to add an nginx ingress controller to a kubernetes cluster.

I don't see any errors when I do the above command.

However, when I try to install a helm configuration, such as:

helm install dai eggplant/dai --version 0.6.5 -f dai.yaml --namespace dai

I am getting an error with my Nginx Ingress Controller:

    W0119 11:58:00.550727   60628 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
Error: Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s": dial tcp 10.108.86.48:443: connect: connection refused

I think this is because of some kind of DNS error. I don't know where the IP 10.108.86.48:443 is coming from or how to find out.

I have also enabled a bunch of ports with firewall-cmd.

    [root@manager-node ~]# sudo firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: ens33
  sources: 
  services: dhcpv6-client ssh
  ports: 6443/tcp 2379-2380/tcp 10250/tcp 10251/tcp 10252/tcp 10255/tcp 443/tcp 30154/tcp 31165/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

However, my nginx ingress pod doesn't seem to start either:

    NAME                                       READY   STATUS              RESTARTS   AGE
ingress-nginx-controller-7bc44b4bb-rwmh2   0/1     ContainerCreating   0          19h

It remains as ContainerCreating for hours.

Any help would be greatly appreciated - I am still new to Kubernetes!

Can you check status of ingress-nginx-admission-create and ingress-nginx-admission-patch Jobs ? — matt_j

ddulic ddulic · Accepted Answer · 2021-01-19T17:51:40

The issue is that as part of that kubectl apply -f you are also applying a ValidatingWebhookConfiguration (check the applied manifest file).

See Using Admission Controllers | Kubernetes Using Admission Controllers | Kubernetes for more info.

The error you are seeing is because your Deployment is not starting up, and thus the ValidatingWebhook service configured as part of it isn't starting up either, so the Validating Controller in Kubernetes is failing every request.

            - --validating-webhook=:8443
            - --validating-webhook-certificate=/usr/local/certificates/cert
            - --validating-webhook-key=/usr/local/certificates/key

Your pod is most likely not starting for another reason. More information is required to further debug.

I would recommend removing the ValidatingWebhookConfiguration from the applied manfiest.

You can also remove it manually with

kubectl delete ValidatingWebhookCOnfiguration ingress-nginx-admission

(Validating Controllers aren't namespaced)

Kubernetes Ingress Controller: Failed calling webhook, dial tcp connect: connection refused

1 Answers