Flink multi slot TaskManagers best practice

2
votes

Say one is running a cluster set to parallelism > 1. What is the best practice for configuring the number of task slots in a task manager?

  • Run multiple TM with a single task slot per each
  • Run a single/few TM with multiple task slots

We are running a Flink cluster with version 1.6.3. Flink 1.7 release notes states that "Flink now properly supports TaskManagers with multiple slots". Is it recommended to start them with a single slot in previous versions?

1
apache-flinkflink-streaming
I was surprised by that announcement too. We were running multislot taskmanagers on YARN before (since version 1.3) and never run into problems even with more complex jobs with bigger states (~50-100 GB). - TobiSH

1 Answers

3
votes

With Flink 1.5 and 1.6, except when running with legacy mode turned on (via mode: legacy), the allocation of TaskManagers with multiple slots is not fully supported and it is recommended to set taskmanager.numberOfTaskSlots: 1. Now that the refactoring of the runtime known as FLIP-6 has been completed, this restriction has been lifted.