Kubernetes Ingress and Node Failure

Question

I am trying to understand Kubernetes High Availability for calls that originate outside the cluster. (For my new On-Premises Kubernetes cluster)

For Example: a desktop application needs to call a service hosted in Kubernetes.

I see two primary ways this can happen, NodePort and Ingress.

NodePort

What will happen during a node failure with NodePort seem fairly obvious. Any calls that are using the newly offline node as the node they are addressing will fail. If you are using a NodePort you need to make sure you have some way of detecting that the node you are using is down, and moving traffic to another node. (And recovering any lost traffic that happened before you detected the downtime.)

This seems like the job of a load balancer. It can do health monitoring and send the traffic to nodes that are up.

Ingress

I am less sure about how an Ingress Controller and an Ingress resource work. It seems like there is a bit of magic here. Does the Ingress controller use NodePort under the hood? I see a lot of assumptions around a load balancer balancer that are not explained in the posts I read. Is it expected that there will be a load balancer in place outside the cluster?

In which case what is the difference between NodePort with a load balancer and Ingress with a load balancer? The load balancer presents a single URL to the calling systems and then it does its best to make the call get to a Kubernetes Node that is up (either via NodePort or Ingress Controller). (Which makes me ask why both are present if both really require a load balancer to hide the node that gets called.)

But I am again confused because you use a single URL to send to an Ingress resource, so how is the node decided by the load balancer?

As I look at this, it seems that Kubernetes only has High Availability inside the cluster. From outside the cluster, it is the job of the caller to ensure that the node they will be addressing, either via your own ping system, or a load balancer.

Does Kubernetes have any features to help with node down times? From the point of view of a desktop application addressing the cluster, not from the point of view of the pods keeping the service up.

mmking mmking · Accepted Answer · 2021-02-25T05:17:16

There's a couple pieces to this question - answering them in turn. The default service types in Kubernetes are ClusterIP, NodePort, and LoadBalancer.

What's a service?

In Kubernetes, a Service is an abstraction which defines a logical set of Pods and a policy by which to access them (sometimes this pattern is called a micro-service).

Basically, it's a logical load balancer in front of a collection of pods, determined by pod labels.

How does a ClusterIP work?

ClusterIP is an internal service, which means the IP address assigned to the service is internal to the cluster. It works by some magic in kube-proxy (read the documentation if you'd like to know more). Dead pods are automatically removed from the load balancing.

How does NodePort work?

NodePort is an external service, which works by opening ports in the host machine. A ClusterIP is automatically created (reference), and through more kube-proxy magic, traffic hitting the NodePort is sent to the ClusterIP, which is in turn sent to a backend pod as appropriate.

You are correct that if the node whose NodePort you are hitting goes down, you won't be able to reach your application.

   traffic
      |
      v
--------------    --------------    --------------
|            |    |            |    |            |
|  NodePort  |    |  NodePort  |    |  NodePort  |
|            |    |            |    |            |
--------------    --------------    --------------

How does LoadBalancer work?

LoadBalancer is an external service, which works by load balancing traffic across the open NodePorts on each node. This means under the cover, when you create a LoadBalancer, a NodePort is created. Generally, this works with a cloud provider: a cloud provider load balancer resource is created that routes traffic to the NodePorts on each node. Health checks are built into the cloud provider load balancer to determine which nodes are up.

                     traffic
                        |
                        v
              --------------------
              |                  |
              |  Cloud Provider  |
              |  Load Balancer   |
              |                  |
              --------------------
      |-----------------|------------------|
      v                 v                  v
--------------    --------------    --------------
|            |    |            |    |            |
|  NodePort  |    |  NodePort  |    |  NodePort  |
|            |    |            |    |            |
--------------    --------------    --------------

How does an ingress controller work?

An ingress controller is (generally) a software component running on the cluster which provides L7 routing (as opposed to L4 routing provided by LoadBalancer). Usually, this will automatically create a LoadBalancer service because the ingress controller is a set of pods running inside the cluster.

Flow occurs something like this:

Image borrowed from here.

Note the green arrows in front of the pods actually indicate load balancing with ClusterIPs.

What should I use for external traffic?

Use LoadBalancer or ingress controller. Don't use NodePort for exactly the reason you described: you have no way of knowing the node is down.