kafka-consumer-api

I'm trying to test run a single Kafka node with 3 brokers & zookeeper. I wish to test using the console tools. I run the producer as such: kafka-console-producer --broker-list localhost:9092,localhost:9093,localhost:9094 --topic testTopic Then I run the consumer as such: kafka-console-consumer --zookeeper localhost:2181 --topic testTopic --from-beginning And I can enter messages in the producer and see them in the consumer, as expected. However , when I run the updated version of the consumer using bootstrap-server, I get nothing. E.g kafka-console-consumer --bootstrap-server localhost:9092

When a new consumer/brorker is added or goes down, Kafka triggers a rebalance operation. Is Kafka Rebalancing a blocking operation. Are Kafka consumers blocked while a rebalancing operation is in progress? Depends on what you mean by "blocked". If you mean "are existing connections closed when rebalance is triggered" then the answer is yes. The current Kafka's rebalancing algorithm is unfortunately imperfect. Here is what is happening during consumer rebalance. Assume we have a topic with 10 partitions (0-9), and one consumer (lets name it consumer1 ) consuming it. When a second consumer

I recently started learning Kafka and end up with these questions. What is the difference between Consumer and Stream? For me, if any tool/application consume messages from Kafka is a consumer in the Kafka world. How Stream is different as this also consumes from or produce messages to Kafka? and why is it needed as we can write our own consumer application using Consumer API and process them as needed or send them to Spark from the consumer application? I did Google on this, but did not get any good answers for this. Sorry if this question is too trivial. Update April 09, 2018 : Nowadays you

There is only few serializer available like, org.apache.kafka.common.serialization.StringSerializer org.apache.kafka.common.serialization.StringSerializer How can we create our own custom serializer ? Here you have an example to use your own serializer/deserializer for the Kafka message value. For Kafka message key is the same thing. We want to send a serialized version of MyMessage as Kafka value and deserialize it again into a MyMessage object at consumer side. Serializing MyMessage in producer side. You should create a serializer class that implements org.apache.kafka.common.serialization

I am trying to load a simple text file instead of standard input in Kafka. After downloading Kafka, I performed the following steps: Started zookeeper: bin/zookeeper-server-start.sh config/zookeeper.properties Started Server bin/kafka-server-start.sh config/server.properties Created a topic named "test": bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test Ran the Producer: bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test Test1 Test2 Listened by the Consumer: bin/kafka-console-consumer.sh --zookeeper localhost:2181 -

I am using kafka 0.10.1.1 and confused with the following 3 properties. heartbeat.interval.ms session.timeout.ms max.poll.interval.ms heartbeat.interval.ms - This was added in 0.10.1 and it will send heartbeat between polls. session.timeout.ms - This is to start rebalancing if no request to kafka and it gets reset on every poll. max.poll.interval.ms - This is across the poll. But, when does kafka starts rebalancing? Why do we need these 3? What are the default values for all of them? Thanks Assuming we are talking about Kafka 0.10.1.0 or upwards where each consumer instance employs two threads