问题
I have been exploring Apache Flink for a few days, and I have some doubts about the concept of Task Slot. Although several questions have been asked about it, there is a point I don't get.
I am using a toy application for testing, running a local cluster. I have disabled operator chaining
I know from docs that slots allow for memory isolation and not CPU isolation. Reading the docs, it seems that a Task Slot is a Java thread.
1) When I deploy my application with parallelism=1, all the operators' subtasks are deployed in the same slot. However, if I print the current thread ID from the open() method of AbstractStreamOperator, I see different IDs for different subtasks. So, aren't they sharing the same thread (i.e., the slot?).
2) If I change the parallelism from 1 to 3, I need 3 slots in order for the application to be re-deployed correctly. Documentation confirms that the number of slots limits the parallelism I can have. But why can I have subtasks of different operators in the same slot, while I cannot have subtasks of the same operator in the same slot?
Thanks for any explanation!
回答1:
The idea of slots is to slice the available resources up into smaller parts. The available managed memory is evenly distributed among all slots. CPU cycles and JVM heap memory are not properly isolated wrt slots.
In each slot you can deploy one or more Tasks. A Flink Task is executed by a dedicated thread. Thus, you can have multiple threads running in the same slot if you have multiple Tasks deployed to it.
A Task represents a parallel instance of a single Flink operator or of multiple operators if they are chainable. Chaining is not always possible or desired but if applied it will fuse operators so that they are executed by the same Task thread. This is usually more efficient since there are fewer context switches and no handing over of records to a different thread.
In order to improve resource utilization (especially for Tasks which need little resources) and to make the reasoning about how many slots you need to run a Flink program easier, Flink supports slot sharing. Slot sharing means that parallel instances of different operators can be deployed to the same slot. Due to this feature, Flink creates as long pipelines of different operators as possible and deploys them to the same slot. This has also the nice effect that you increase co-location of producers with their respective consumers. Due to this feature, users know that they only need to provide as many slots as the maximum parallelism of all operators of ones topology.
However, since you still want to distribute parallel instances of an operator across all available TaskExecutors, Flink does not support to deploy parallel instances of the same operator to the same slot. If you want to do this, then you should simply reduce the parallelism of the respective operator to 1.
来源：https://stackoverflow.com/questions/55590809/can-i-have-multiple-subtasks-of-an-operator-in-the-same-slot-in-flink