apache-kafka-streams

问题 I'm doing a hopping window aggregation on a 4 hr window advancing every 5 mins. As the hopping windows are overlapping, I'm getting duplicate keys with different aggregated value. TimeWindows.of(240 * 60 * 1000L).advanceBy(5 * 60* 1000L) How do I eliminate duplicate keys with repeating data or pick only the keys that holds the latest value. 回答1: If I understand you correctly, then this is expected behavior. You are not seeing "duplicate" keys, but you see continuous updates for the same key.

问题 We increased the number of partitions to parallel process the messages as the throughput of the message was high. As soon as we increased the number of partitions all the streams thread which were subscribed to that topic died. We changed the consumer group id then we restarted the application it worked fine. I know that the number of partitions changelog topic of application should be same as source topic. I would like to know the reason behind this. I saw this link - https://issues.apache

问题 So I am trying to get interactive queries working with Kafka streams. I have Zookeeper and Kafka running locally (on windows). Where I use the C:\temp as the storage folder, for both Zookeeper and Kafka. I have setup the topic like this kafka-topics.bat --zookeeper localhost:2181 --create --replication-factor 1 --partitions 1 --topic rating-submit-topic kafka-topics.bat --zookeeper localhost:2181 --create --replication-factor 1 --partitions 1 --topic rating-output-topic Reading I have Done

问题 I'm working with Kafka that save the data into rocksdb. Now I want to have a look at the db keys and values that created by Kafka. I downloaded FastNoSQL and tried but failed. The folder contains: .sst files .log files CURRENT file IDENTITY file LOCK file LOG files MANIFEST files OPTIONS files How can I watch the values? 回答1: Keylord (since version 5.0) can open RocksDB databases. For example here is Kafka stream of Wordcount application: 回答2: For RocksDB db files you can use FastoNoSql. 回答3:

问题 In Kafka Streams, whats the canonical way of producing/writing a stream? In Spark, there is the custom receiver which works as a long running adapter from an arbitrary data source. What is the equivalent in Kafka Streams? To be specific, I'm not asking how to do transforms from one topic to another. The documentation is very clear on that. I want to understand how to write my workers that will be doing the first write in a series of transforms into Kafka. I expect to be able to do builder1.

问题 I have a Kafka producer which is producing messages at high rate (message key is let us say a username and value is his current score in a game). The Kafka consumer is relatively slow in processing the consumed messages. Here my requirement is to show most up-to-date score and avoid showing stale data, with the tradeoff that some scores may never be shown. Essentially for each of the username, I may have hundreds of messages in the same partition, but I always want to read the latest one. A

问题 I have an application that needs to listen to multiple different topics; each topic has separate logic for how the messages are handled. I had thought to use the same kafka properties for each KafkaStreams instance, but I get an error like the one below. Error java.lang.IllegalArgumentException: Assigned partition my-topic-1 for non-subscribed topic regex pattern; subscription pattern is my-other-topic Code (kotlin) class KafkaSetup() { companion object { private val LOG = LoggerFactory

问题 I am using Kafka Streams with Spring Boot. In my use case when I receive customer event from other microservice I need to store in customer materialized view and when I receive order event, I need to join customer and order then store in customer-order materialized view. To achieve this I created persistent key-value store customer-store and updating this when a new event comes. StoreBuilder customerStateStore = Stores.keyValueStoreBuilder(Stores.persistentKeyValueStore("customer"),Serdes

问题 I have a single node kafka broker and simple streams application. I created 2 topics (topic1 and topic2). Produced on topic1 - processed message - write to topic2 Note: For each message produced only one message is written to destination topic I produced a single message. After it was written to topic2, I stopped the kafka broker. After sometime I restarted the broker and produced another message on topic1. Now streams app processed that message 3 times. Now without stopping the broker I

问题 I had a scenario to read the JSON data from my Kafka topic, and by making use of Kafka 0.11 version I need to write Java code for streaming the JSON data present in the Kafka topic.My input is a Json Data containing arrays of Dictionaries. Now my requirement is to get the "text" field, key in dictionary contained in array from the json data and pass all those text tweets to another topic through Kafka Streaming. I wrote code till here. Please help me to parse the data. Java code for streaming