Is one TaskManager with three slots the same as three TaskManagers with one slot in Apache Flink

旧城冷巷雨未停 提交于 2020-06-17 02:53:25

问题


In Flink, as my understanding, JobManager can assign a job to multiple TaskManagers with multiple slots if necessary. For example, one job can be assigned three TaskManagers, using five slots.

Now, saying that I execute one TaskManager(TM) with three slots, which is assigned to 3G RAM and one CPU.

Is this totally the same as executing three TaskManagers, sharing one CPU, and each of them is assigned to 1 G RAM?

case 1
---------------
| 3G RAM      |
| one CPU     |
| three slots |
| TM          |
---------------

case 2
--------------------------------------------|
|              one CPU                      |
|  ------------  ------------ ------------  |
|  | 1G RAM   |  | 1G RAM   | | 1G RAM   |  |
|  | one slot |  | one slot | | one slot |  |
|  | TM       |  | TM       | | TM       |  |
|  ------------  ------------ ------------  |
--------------------------------------------|

回答1:


There are performance and operational differences that pull in both directions.

When running in non-containerized environments, with the RocksDB state backend, it can make sense to have a single TM per machine, with many slots. This will minimize the per-TM overhead. However, the per-TM overhead is not that significant.

On the other hand, running with one slot per TM provides some helpful isolation, and reduces the impact of garbage collection, which is particularly relevant with a heap-based state backend.

With containerized deployments, it is generally recommended to go with one slot per TM until reaching some significant scale, at which point you will want to scale by adding more slots per TM rather than more TMs. The issue is that the checkpoint coordinator needs to coordinate with each TM (but not with each slot), and as the number of TMs gets into the hundreds or thousands, this can become a bottleneck.



来源:https://stackoverflow.com/questions/62326839/is-one-taskmanager-with-three-slots-the-same-as-three-taskmanagers-with-one-slot

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!