What does Traefik do to connections to deleted Pods?

Question

Imagine you have a k8s cluster set up with Traefik as an Ingress Controller.

An HTTP API application is deployed to the cluster (with an ingress resource) that is able to handle SIGTERM and does not exit until all active requests are handled.

Let's say you deploy the application with 10 replicas, get some traffic to it and scale the deployment down to 5 replicas. Those 5 Pods will be pulled out from the matching Service resource.

For those 5 Pods, the application will receive SIGTERM and start the graceful shutdown.

The question is, what will Traefik do with those active connections to the pulled out 5 Pods?

• Will it wait until all the responses are received from the 5 Pods and not send any traffic to them during and after that?
• Will it terminate the ongoing connections to those 5 Pods and forget about them?

Answer 1

Traefik will do the first: it will gracefully let those pending, in-flight requests finish but not forward any further requests to the terminating pods.

To add some technical background: once a pod is deemed to terminate from Kubernetes' perspective, the Endpoints controller (also part of the Kubernetes control plane) will remove its IP address from the associated endpoints object. Traefik watches for updates on the endpoints, receives a notification, and updates its forwarding rules accordingly. So technically, it is not able to forward any further traffic while those final requests will continue to be served (by previously established goroutines from Go's http package).