kafka-consumer-api

AFAIK, The concept of partitions and (consumer) groups in kafka was introduced to implement parallelism. I am working with kafka through python. I have a certain topic, which has (say) 2 partitions. This means, if I start a consumer group with 2 consumers in it, they will be mapped(subscribed) to different partitions. But, using kafka library in python, I came across a weird issue. I started 2 consumers with essentially the same group-ids, and started the threads for them to consume messages. But, every message in the kafka-stream is being consumed by both of them !! This seems ridiculous to

I am using a Kafka producer - consumer model in my framework. The record consumed at the consumer end is later indexed onto the elasticsearch. Here i have a use case where if the ES is down, I will have to pause the kafka consumer until the ES is up, Once it is up, I need to resume the consumer and consume the record from where I last left. I don't think this can be achieved with @KafkaListener. Can anyone please give me a solution for this? I figured out that I need to write my own KafkaListenerContainer for this, but I am not able to implement it correctly. Any help would be much appreciated

To achieve exactly-once processing of messages by Kafka consumer I am committing one message at a time, like below public void commitOneRecordConsumer(long seconds) { KafkaConsumer<String, String> consumer = consumerConfigFactory.getConsumerConfig(); try { while (running) { ConsumerRecords<String, String> records = consumer.poll(1000); try { for (ConsumerRecord<String, String> record : records) { processingService.process(record); consumer.commitSync(Collections.singletonMap(new TopicPartition(record.topic(),record.partition()), new OffsetAndMetadata(record.offset() + 1))); System.out.println(

We have 3 zk nodes cluster and 7 brokers. Now we have to create a topic and have to create partitions for this topic. But I did not find any formula to decide that how much partitions should I create for this topic. Rate of producer is 5k messages/sec and size of each message is 130 Bytes. Thanks In Advance It depends on your required throughput, cluster size, hardware specifications: There is a clear blog about this written by Jun Rao from Confluent: How to choose the number of topics/partitions in a Kafka cluster? Also this might be helpful to have an insight: Apache Kafka Supports 200K

Using kafka-python-1.0.2. If I have a topic with 10 partitions, how do I go about committing a particular partition, while looping through the various partitions and messages. I just cant seem find an example of this anywhere, in the docs or otherwise From the docs, I want to use: consumer.commit(offset=offsets) Specifically, how do I create the partition and OffsetAndMetadata dictionary required for offsets (dict, optional) – {TopicPartition: OffsetAndMetadata}. I was hoping the function call would just be something like: consumer.commit(partition, offset) but this does not seem to be the

I have setup a kafka zookeeper and 3 brokers on one machine on ec2 with ports 9092..9094 and am trying to consume the topic content from another machine. The ports 2181 (zk), 9092, 9093 and 9094 (servers) are open to the consumer machine. I can even do a bin/kafka-topics.sh --describe --zookeeper 172.X.X.X:2181 --topic remotetopic which gives me Topic:remotetopic PartitionCount:1 ReplicationFactor:3 Configs: Topic: remotetopic Partition: 0 Leader: 2 Replicas: 2,0,1 Isr: 2,0,1 Blockquote However when i do bin/kafka-console-consumer.sh --zookeeper 172.X.X.X:2181 --from-beginning --topic