kafka-consumer-api

问题 I am trying to create a Kafka topology and break it down into more readable. I have a stream that I group by keys, and then I am trying to window it like so: SessionWindowedKStream<byte[], byte[]> windowedTable = groupedStream.windowedBy(SessionWindows.with(Duration.ofSeconds(config.joinWindowSeconds)).grace(Duration.ZERO)); KTable<Windowed<byte[]>, byte[]> mergedTable = windowedTable .reduce((aggregateValue, newValue) -> { try { Map<String, String> recentMap = MAPPER.readValue(new String

问题 I am trying to create a Kafka topology and break it down into more readable. I have a stream that I group by keys, and then I am trying to window it like so: SessionWindowedKStream<byte[], byte[]> windowedTable = groupedStream.windowedBy(SessionWindows.with(Duration.ofSeconds(config.joinWindowSeconds)).grace(Duration.ZERO)); KTable<Windowed<byte[]>, byte[]> mergedTable = windowedTable .reduce((aggregateValue, newValue) -> { try { Map<String, String> recentMap = MAPPER.readValue(new String

问题 Is there any known issue with kakfa-broker in managing the offsets? Bcz, problem which we are facing is when we try to restart of kafka-consumer(i.e, app restart) sometimes all the offset are reset to 0. Completely clueless on why are consumers not able to start from the last commited offset. We are eventually facing this issue in prod wherein the whole q events are replayed again : spring-boot version -- 2.2.6 release spring-kafka - 2.3.7 release kafka-client -2.3.1 apache-kafka - kafka_2.12

问题 If I create 2 kafka consumer instances passing same properties subscribe on same topic Will these 2 Consumer instances (at diff group Id), have similar partition structures, or could be different ? i.e, if I do .assignment() will I get same result at both My actual problem statement, where I will be using this validation In my application, I am attaining offset of broker, at a particular state(This is being done through my 1st kafka consumer object). Later, I am creating the 2nd kafka

问题 I want suggestions of alternatives to Spring-Kafka. I have used Kafka integrated in Spring in my application. I want to explore any alternative libraries available. Would be good if comparative analysis is share between libraries. 回答1: In addition to the plain Java API ... Put site:github.com + kafka in your favorite search engine... wix/greyhound (seems pretty simple, and based on ZIO) zio-kafka (ZIO is cool) ExpediaGroup/rhapsody (Worth a look) dropwizard/dropwizard-kafka (my personal 2nd

问题 I have a kafka setup that includes a jmx exporter to prometheus. I'm looking for a metric, that gives the offset lag based on topic and groupid. I'm running kafka 2.2.0. Some resources online point to a metric called kafka.consumer , but I have no such metric in my setup. From my jmxterminal: $>domains #following domains are available JMImplementation com.sun.management java.lang java.nio java.util.logging jdk.management.jfr kafka kafka.cluster kafka.controller kafka.coordinator.group kafka

问题 I have a kafka setup that includes a jmx exporter to prometheus. I'm looking for a metric, that gives the offset lag based on topic and groupid. I'm running kafka 2.2.0. Some resources online point to a metric called kafka.consumer , but I have no such metric in my setup. From my jmxterminal: $>domains #following domains are available JMImplementation com.sun.management java.lang java.nio java.util.logging jdk.management.jfr kafka kafka.cluster kafka.controller kafka.coordinator.group kafka

问题 According to Kafka documentation; The new Java Consumer now supports heartbeating from a background thread. There is a new configuration max.poll.interval.ms which controls the maximum time between poll invocations before the consumer will proactively leave the group (5 minutes by default). The value of the configuration request.timeout.ms must always be larger than max.poll.interval.ms because this is the maximum time that a JoinGroup request can block on the server while the consumer is

问题 According to Kafka documentation; The new Java Consumer now supports heartbeating from a background thread. There is a new configuration max.poll.interval.ms which controls the maximum time between poll invocations before the consumer will proactively leave the group (5 minutes by default). The value of the configuration request.timeout.ms must always be larger than max.poll.interval.ms because this is the maximum time that a JoinGroup request can block on the server while the consumer is

问题 How to use li-apache-kafka-clients in spring boot app to send large message (above 1MB) from Kafka producer to Kafka Consumer? Below is the GitHub link of li-apache-kafka-clients: https://github.com/linkedin/li-apache-kafka-clients I have imported .jar file of li-apache-kafka-clients and put the below configuration for producer: props.put("large.message.enabled", "true"); props.put("max.message.segment.bytes", 1000 * 1024); props.put("segment.serializer", DefaultSegmentSerializer.class