I have a statefulset which has a nodeSelector:
nodeSelector:
app: licensed
When I assign the node with app: licensed, I can see the pod is schedule on a specific node.
But when I remove the label from the node, I don't see k8s remove the pod from that node. I have to explicitly delete the pod.
Is it a kubernetes feature? Or did I use the nodeSelector correctly?
The thing you're looking for is likely taints and tolerations if you want only designated things to run on designated Nodes. The nodeSelector: you used is used in the scheduling decision, but is not used in the reconciliation loop (as you experienced)
There is more fine grained control via the effect: NoSchedule, which as it implies will keep a Pod merely from being scheduled if it doesn't tolerate the taint, and effect: NoExecute which is closer to what you're asking for where if a taint appears on a Node, and the Pod doesn't tolerate it, the Pod will be evicted and not rescheduled upon any such tainted Node. Modern versions of kubernetes allow even finer grained control via the PreferNoSchedule and the opposite end of that with tolerationSeconds to give some padding around how soon any such eviction may happen
