Scale kafka consumers and distribute them across multiple nodes of app server

折月煮酒 提交于 2021-02-11 14:16:10

问题


We have java app which runs on app server.

We have topic topic1 with 5 partitions. We have created 5 consumer threads in a consumer group "consumergroup1". One consumer thread per partition. They are running on appserver server one.

Now we bring up one more app server servertwo on another node.

We want to run some of consumers in same group "consumergroup1" for topic1 on new server. Basically distribute consumer threads between two app servers for same consumer group.

If we bring up consumers on server two for consumergroup1 they would be redundant as already 5 consumer threads are running on server one which is equal to total partitions in topic1

Can anyone suggest how to redistribute consumer threads across multiple appservers nodes. The app server nodes are scaled based on demand

来源:https://stackoverflow.com/questions/62662476/scale-kafka-consumers-and-distribute-them-across-multiple-nodes-of-app-server

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!