Kafka connect internals - how connectors and tasks got deployed around the connect cluster

橙三吉。 提交于 2019-12-11 15:27:59

问题


I use Kafka Connect for different purposes and it's working fine. It's more a curiosity question. Trying to figure out reading the code might take some time, so I'm asking here.. ( but I'll try read Kafka code anyway..)

I know a Connector is the one responsible for giving/updating configurations for the tasks, but what it is exactly ? Is it some piece of code that will run over the Connect cluster ? If yes, I imagine a worker did initiate it, but it does it arbitrary on one worker JVM ? Besides, is there only one instance of the connector , or is there multiple instance running at the same time ?

Same kind of questions for the Tasks, when we submit a connector to a Connect cluster (workers), how do they reach a consensus of " ok we will run this tasks in this worker, and this task in this another worker" ? How do they know about how many tasks are actually running or not ?

来源:https://stackoverflow.com/questions/58026235/kafka-connect-internals-how-connectors-and-tasks-got-deployed-around-the-conne

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!