we'd like to scale in some of the running instances on which many kubernetes pods are running. So, we are going to gracefully stop the pods by using graceful period according to the official document termination-of-pods. I have read many blog posts and official document, they all tells how to gracefully terminate pod with graceful period. But they do not say how to determine how long the graceful period would be better.
Let's say, for example, a container in a pod may serves for thousands of requests in a time period and it will spend more than 30s to complete all request. I think in this case it would be a bad idea to set graceful period to 30s, because some of the request would be lost. However, when the user load is down and the same container in the same pod serves for only dozens of request in other time period and it only spend 5s to complete all, in this case 30s for graceful period would be too long.
That's my consideration. So, my question is as follows.
1. Is there any best practice to determine how long the graceful period is better?
2. Is there any approach to check if the processing request is completed in a container and then gracefully terminate pod?
3. Can I extend the initial graceful period after sending the termination command to a pod?
Thanks in advance.
