kafka-consumer-api

I was going through the Consumer Config for Kafka. https://kafka.apache.org/documentation/#newconsumerconfigs what are the parameters that will trigger a rebalance ?. For instance the following parameter will ?. Any other parameters which we need to change or will defaults suffice connections.max.idle.ms Close idle connections after the number of milliseconds specified by this config. long 540000 medium Also we have three different topics Is it a bad idea to have the same Consumer Group (Same ID) consuming from multiple topics. Assuming if the above scenario is valid (not necessarily the best

I have been using the python-kaka module to consume from a kafka broker. I want to consume from the same topic with 'x' number of partitions in parallel. The documentation has this : # Use multiple consumers in parallel w/ 0.9 kafka brokers # typically you would run each on a different server / process / CPU consumer1 = KafkaConsumer('my-topic', group_id='my-group', bootstrap_servers='my.server.com') consumer2 = KafkaConsumer('my-topic', group_id='my-group', bootstrap_servers='my.server.com') Does this mean I can create a separate consumer for each process that I spawn? Also, will there be an

I have 3 node(nodes0,node1,node2) Kafka cluster(broker0, broker1, broker2) with replication factor 2 and Zookeeper(using zookeeper packaged with Kafka tar) running on a different node (node 4). I had started broker 0 after starting zookeper and then remaining nodes. It is seen in broker 0 logs that it is reading __consumer_offsets and seems they are stored on broker 0. Below are sample logs: Kafka Version: kafka_2.10-0.10.2.0 2017-06-30 10:50:47,381] INFO [GroupCoordinator 0]: Loading group metadata for console-consumer-85124 with generation 2 (kafka.coordinator.GroupCoordinator) [2017-06-30

I am using Spring Kafka first time and I am not able to use Acknowledgement.acknowledge() method for manual commit in my consumer code as mentioned here https://docs.spring.io/spring-kafka/reference/html/_reference.html#committing-offsets . Mine is spring-boot application. If I am not using manual commit process than my code is working fine. But when I use Acknowledgement.acknowledge() for manual commit it shows error related to bean. Also If I am not using manual commit properly please suggest me the right way to do it. Error message: *************************** APPLICATION FAILED TO START **

When trying to consume from Kafka using the high-level consumer (using a completely new consumer group), the consumer never starts running. When I switch the logging level to debug, I can see the following two lines repeating themselves over and over: DEBUG [AbstractCoordinator] 09:43:51.192: Sending coordinator request for group CompletelyNewConsumerGroupThatIneverUsedBefore to broker 172.30.3.61:30000 (id: 1043 rack: null) DEBUG [AbstractCoordinator] 09:43:51.306: Received group coordinator response ClientResponse(receivedTimeMs=1487666631306, disconnected=false, request=ClientRequest

I have a producer which writes messages to a topic/partition. To maintain ordering, i would like to go with single partition and I want 12 consumers to read all the messages from this single partition(no consumer group, all the messages should go to all consumers). Is this achievable? I read some forums that only one consumer can read per partition. You may use SimpleConsumer to achieve exactly what you are asking - no consumer groups, all consumers can read a single partition. However this approach means you have to handle offset storing and broker failure handling yourself. Another option is