I am currently trying to scale automatically a deployment when the readiness probe hass failed for its pods currently running.
A pod is IDLE until a POST request is sent to it and while it is processing the request, it is not answering any other request.
To know when a processing is in progress, I created an endpoint returning TRUE if the pod is IDLE, FALSE otherwise.
I configure the readiness probe to query this endpoint to mark it as unavailable when a processing is in progress (and to mark it back to available when it is not processing anymore).
By default I have a limited pool of pods (like 5) that can answer requests.
But I still want to be able to send another POST with other parameters to trigger another processing when all my 5 pods are unavailable.
So, when the readiness probe fails for all pods, I want to scale the deployment in order to have other pods availbel to answer requests.
The issue here is that I did not find how to do such a thing with K8S or if this is even possible. Is there someone who could help me on this?
An alternative would be to create a 'watcher' pod who would watch all the readiness probe for a given deployement and when I fails for all pods, the watch would be in charge of scaling the deployement.
But this alternative implies development that I would like to avoid if it is natively possible to do in K8S.
Thank you :)
