Flink task scheduling for operators with huge state

Question

Let's assume a setting where we have one or more operators connected to a keyed stream, and where we expect numerous keys and significant partitioned state (in terms of size) per key.

Is it possible to influence on which Task Managers Flink would schedule each of these operator instances? In a setting with n Task Managers and m operator instances, the worst case would be that multiple (if not all) of these m instances end up sitting on only a few of the n Task Managers, therefore exhausting the available resources.

Answer 1

At the moment, Flink treats every Task and Slot equally. This means that every Task should be deployable into every Slot. This does not only simplify the scheduling logic significantly but it also prevents the user from shooting himself into the foot by specifying unsatisfiable scheduling constraints. The cost is that the user cannot control how Tasks are deployed across a set of available slots and TaskManagers.

The community is currently working on improving Flink's scheduling capabilities by making Flink's scheduler pluggable. With this change we intend to enable more control over how Flink deploys Tasks.