Change Kubernetes nginx-ingress-controller ports

3
votes

I installed Minikube v1.3.1 on my RedHat EC2 instance for some tests.

Since the ports that the nginx-ingress-controller uses by default are already in use, I am trying to change them in the deployment but without result. Could please somebody advise how to do it?

How do I know that the port are already in Use?

When I listed the system pods using the command kubectl -n kube-system get deployment | grep nginx, I get:

nginx-ingress-controller 0/1 1 0 9d

meaning that my container is not up. When I describe it using the command kubectl -n kube-system describe pod nginx-ingress-controller-xxxxx I get:

Type Reason Age From
Message ---- ------ ----
---- ------- Warning FailedCreatePodSandBox 42m (x163507 over 2d1h) kubelet, minikube (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to start sandbox container for pod "nginx-ingress-controller-xxxx": Error response from daemon: driver failed programming external connectivity on endpoint k8s_POD_nginx-ingress-controller-xxxx_kube-system_...: Error starting userland proxy: listen tcp 0.0.0.0:443: bind: address already in use

Then I check the processes using those ports and I kill them. That free them up and the ingress-controller pod gets deployed correctly.

What did I try to change the nginx-ingress-controller port?

kubectl -n kube-system get deployment | grep nginx

> NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
> nginx-ingress-controller   0/1     1            0           9d

kubectl -n kube-system edit deployment nginx-ingress-controller

The relevant part of my deployment looks like this:

name: nginx-ingress-controller
        ports:
        - containerPort: 80
          hostPort: 80
          protocol: TCP
        - containerPort: 443
          hostPort: 443
          protocol: TCP
        - containerPort: 81
          hostPort: 81
          protocol: TCP
        - containerPort: 444
          hostPort: 444
          protocol: TCP
        - containerPort: 18080
          hostPort: 18080
          protocol: TCP

Then I remove the subsections with port 443 and 80, but when I rollout the changes, they get added again.

Now my services are not reachable anymore through ingress.

3
nginxkubernetesminikubekubernetes-ingress
Hello AR1, Can you please share the complete NGINX deployment YAML? It would be helpful to completely understand the scenario. On the other hand, are you using the K8s documentation[1] or other guide? [1] kubernetes.io/docs/tasks/access-application-cluster/… – Luis Javier Alvarez Rodriguez
Hi Luis, thanks for your answer. I didn't post the full YAML because of its length. The part that I posted is the only difference with the default one. I followed the documentation already and ingress is working fine unless I change the NGINX port as specified in the question. – AR1

3 Answers

1
votes

Please note that minikube ships with addon-manager, which role is to keep an eye on specific addon template files (default location: /etc/kubernetes/addons/) and do one of two specific actions based on the label's value of managed resource:

addonmanager.kubernetes.io/mode

  1. addonmanager.kubernetes.io/mode=Reconcile

Will be periodically reconciled. Direct manipulation to these addons through apiserver is discouraged because addon-manager will bring them back to the original state. In particular

  1. addonmanager.kubernetes.io/mode=KeepOnly

Will be checked for existence only. Users can edit these addons as they want.

So to keep your customized version of default Ingress service listening ports, please change first the Ingress deployment template configuration to KeepOnly on minikube VM.

0
votes

Basically, minikube bootstraps Nginx Ingress Controller as the separate addon, thus as per design you might have to enable it in order to propagate the particular Ingress Controller's resources within minikube cluster.

Once you enabled some specific minikube Addon, Addon-manager creates template files for each component by placing them into /etc/kubernetes/addons/ folder on the host machine, and then spin up each manifest file, creating corresponded K8s resources; furthermore Addon-manager continuously inspects the actual state for all addon resources synchronizing K8s target resources (service, deployment, etc.) according to the template data.

Therefore, you can consider modifying Ingress addon template data throughout ingress-*.yaml files under /etc/kubernetes/addons/ directory, propagating the desired values into the target k8s objects; it may takes some until K8s engine reflects the changes and re-spawns the relative ReplicaSet based resources.

0
votes

Well, I think you have to modify the Ingress which refer to the service you're trying to expose on custom port.

This can be done with custom annotation. Here is an example for your port 444:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: myservice
  namespace: mynamespace
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.org/listen-ports-ssl: "444"

spec:
  tls:
  - hosts:
    - host.org
    secretName: my-host-tls-cert
  rules:
  - host: host.org
    http:
      paths:
      - path: /
        backend:
          serviceName: my-service
          servicePort: 444