Is one TaskManager with three slots the same as three TaskManagers with one slot in Apache Flink

1
votes

In Flink, as my understanding, JobManager can assign a job to multiple TaskManagers with multiple slots if necessary. For example, one job can be assigned three TaskManagers, using five slots.

Now, saying that I execute one TaskManager(TM) with three slots, which is assigned to 3G RAM and one CPU.

Is this totally the same as executing three TaskManagers, sharing one CPU, and each of them is assigned to 1 G RAM?

case 1
---------------
| 3G RAM      |
| one CPU     |
| three slots |
| TM          |
---------------

case 2
--------------------------------------------|
|              one CPU                      |
|  ------------  ------------ ------------  |
|  | 1G RAM   |  | 1G RAM   | | 1G RAM   |  |
|  | one slot |  | one slot | | one slot |  |
|  | TM       |  | TM       | | TM       |  |
|  ------------  ------------ ------------  |
--------------------------------------------|
1
parallel-processingarchitectureapache-flink
Not really, because there is a overhead of the JVM memory that TM is taking... i guess in cpu too. The effeicient is that u'll take more easy gc's and not big ones i guess. - ShemTov

1 Answers

2
votes

There are performance and operational differences that pull in both directions.

When running in non-containerized environments, with the RocksDB state backend, it can make sense to have a single TM per machine, with many slots. This will minimize the per-TM overhead. However, the per-TM overhead is not that significant.

On the other hand, running with one slot per TM provides some helpful isolation, and reduces the impact of garbage collection, which is particularly relevant with a heap-based state backend.

With containerized deployments, it is generally recommended to go with one slot per TM until reaching some significant scale, at which point you will want to scale by adding more slots per TM rather than more TMs. The issue is that the checkpoint coordinator needs to coordinate with each TM (but not with each slot), and as the number of TMs gets into the hundreds or thousands, this can become a bottleneck.