Why does kafka producer take a broker endpoint when being initialized instead of the zk

Question

If I have multiple brokers, which broker should my producer use? Do I need to manually switch the broker to balance the load? Also why does the consumer only need a zookeeper en

Answer 1

which broker should my producer use?
Do I need to manually switch the broker to balance the load?

Kafka runs on cluster, meaning set of nodes, so while producing anything you need to tell him the LIST of brokers that you've configured for your application, below is a small note taken from their documentation.

“metadata.broker.list” defines where the Producer can find a one or more Brokers to determine the Leader for each topic. This does not need to be the full set of Brokers in your cluster but should include at least two in case the first Broker is not available. No need to worry about figuring out which Broker is the leader for the topic (and partition), the Producer knows how to connect to the Broker and ask for the meta data then connect to the correct Broker.

Hope this clear some of your confusion

Also why does the consumer only need a zookeeper endpoint instead of a broker endpoint

This is not technically correct, as there are two types of APIs available, High level and Low level consumer.

The high level consumer basically takes care of most of the thing like leader detection, threading issue, etc. but does not provide much control over messages which exactly the purpose of using the other alternatives Simple or Low level consumer, in which you will see that you need to provide the brokers, partition related details.

So Consumer need zookeeper end point only when you are going with the high level API, in case of using Simple you do need to provide other information