K8s scheduling timeout

Question

I have cluster-autoscaler integrated with K8s and worker nodes are scaled from 0. I want to not schedule more than 2 pods per node. I've set pod limit on kubelet level. When I am running 5 jobs in parallel with 4 pods limit It scales up 3 nodes , but It's trying to schedule only on 2 nodes and one pod in going down due limit. Is there any scheduling limit parameter in K8s? To schedule pods only after specific time (some sleep parameter)? We need to wait when all workers become ready

Answer 1

It's not possible with the default scheduler that kubernetes comes with. You need to implement custom scheduler and write the logic in the scheduler to cater to this use case.

Answer 2

We need to wait when all workers become ready

Seems to me you need to use readiness probes on your pods