kafka-consumer-api | 易学教程

Kafka-consumer. commitSync vs commitAsync

阅读更多关于 Kafka-consumer. commitSync vs commitAsync

问题 The quote from https://www.safaribooksonline.com/library/view/kafka-the-definitive/9781491936153/ch04.html#callout_kafka_consumers__reading_data_from_kafka_CO2-1 The drawback is that while commitSync() will retry the commit until it either succeeds or encounters a non-retriable failure, commitAsync() will not retry. This phrase is not clear to me. I suppose that consumer sends commit request to broker and in case if the broker doesn't respond within some timeout it means that the commit

Kafka consumer fetching metadata for topics failed

阅读更多关于 Kafka consumer fetching metadata for topics failed

I am attempting to write a Java client for a third party's Kafka and ZooKeeper servers. I am able to list and describe topics, but when I attempt to read any, a ClosedChannelException is raised. I reproduce them here with the command line client. $ bin/kafka-console-consumer.sh --zookeeper 255.255.255.255:2181 --topic eventbustopic [2015-06-02 16:23:04,375] WARN Fetching topic metadata with correlation id 0 for topics [Set(eventbustopic)] from broker [id:1,host:SOME_HOST,port:9092] failed (kafka.client.ClientUtils$) java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send

Kafka Consumer Rebalancing Algorithm

阅读更多关于 Kafka Consumer Rebalancing Algorithm

Can someone please tell me what the rebalancing algorithm is for Kafka consumers? I would like to understand how partition count and consumer threads affect this. Thank you, Ok so there are 2 rebalancing algorithms at the moment - Range and RoundRobin . They are also called Partition Assignment Strategies. For the simplicity assume we have a topic T1 with 10 partitions and we also have 2 consumers with different configurations (for the example to be clearer) - C1 with num.streams set to 1 and C2 with num.streams set to 2 . Here's how that would work with Range strategy: Range lays out

Kafka only subscribe to latest message?

阅读更多关于 Kafka only subscribe to latest message?

Sometimes(seems very random) Kafka sends old messages. I only want the latest messages so it overwrite messages with the same key. Currently it looks like I have multiple messages with the same key it doesn't get compacted. I use this setting in the topic: cleanup.policy=compact I'm using Java/Kotlin and Apache Kafka 1.1.1 client. Properties(8).apply { val jaasTemplate = "org.apache.kafka.common.security.scram.ScramLoginModule required username=\"%s\" password=\"%s\";" val jaasCfg = String.format(jaasTemplate, Configuration.kafkaUsername, Configuration.kafkaPassword) put(ConsumerConfig

Kafka Connect JDBC sink connector not working

阅读更多关于 Kafka Connect JDBC sink connector not working

问题 I am trying to use Kafka Connect JDBC sink connector to insert data into Oracle but it is throwing an error . I have tried with all the possible configurations of the schema. Below is the examples . Please suggest if I am missing anything below are my configurations files and errors. Case 1- First Configuration internal.value.converter.schemas.enable=false . so I am getting the [2017-08-28 16:16:26,119] INFO Sink task WorkerSinkTask{id=oracle_sink-0} finished initialization and start (org

Does Kafka support request response messaging

阅读更多关于 Does Kafka support request response messaging

问题 I am investigating Kafka 9 as a hobby project and completed a few "Hello World" type examples. I have got to thinking about Real World Kafka applications based on request response messaging in general and more specifically how to link a Kafka request message to its response message. I was thinking along the lines of using a generated UUID as the request message key and employ this request UUID as the associated response message key. Much the same type of mechanism that WebSphere MQ has

Why does a Kafka consumer take a long time to start consuming?

阅读更多关于 Why does a Kafka consumer take a long time to start consuming?

问题 We start a Kafka consumer, listening on a topic which may not yet be created (topic auto creation is enabled though). Not long thereafter a producer is publishing messages on that topic. However, it takes some time for the consumer to notice this: 5 minutes to be exact. At this point the consumer revokes its partitions and rejoins the consumer group. Kafka re-stabilizes the group. Looking at the time-stamps of the consumer vs. kafka logs, this process is initiated at the consumer side. I

Read Kafka topic in a Spark batch job

阅读更多关于 Read Kafka topic in a Spark batch job

I'm writing a Spark (v1.6.0) batch job which reads from a Kafka topic. For this I can use org.apache.spark.streaming.kafka.KafkaUtils#createRDD however, I need to set the offsets for all the partitions and also need to store them somewhere (ZK? HDFS?) to know from where to start the next batch job. What is the right approach to read from Kafka in a batch job? I'm also thinking about writing a streaming job instead, which reads from auto.offset.reset=smallest and saves the checkpoint to HDFS and then in the next run it starts from that. But in this case how can I just fetch once and stop

Kafka multiple partition ordering

阅读更多关于 Kafka multiple partition ordering

问题 I am aware that it is not possible to order multiple partitions in Kafka and that partition ordering is only guaranteed for a single consumer within a group (for a single partition). However with Kafka Streams 0.10 is it now possible to achieve this? If we use the timestamp feature so that each message in each partition maintains the order, at the consumer side, lets say with Kafka Streams 0.10 is this now possible? Assuming we receive all messages could we not sort all the partitions based

How can I get the LATEST offset of a kafka topic?

阅读更多关于 How can I get the LATEST offset of a kafka topic?

问题 I'm writing a kafka consumer using Java. I want to keep the real time of the message, so if there are too many messages waiting for consuming, such as 1000 or more, I should abandon the unconsumed messages and start consuming from the latest offset. For this problem, I try to compare the last committed offset and the latest offset of a topic(only 1 partition), if the difference between these two offsets is larger than a certain amount, I will set the latest offset of the topic as next offset