apache-kafka-streams

问题 In KafkaStream, when implementing a ValueTransformer or ValueTransformerWithKey , on transform() call, I schedule a new Punctuator. When method punctuate() of the Punctuator is executed I want it to forward an event downstream using the context instance. However, the context instance seems not defined when part of a DSL topology. Any clue on how to do this with a Transformer ? Using the same logic in a Processor, implementing the low-level processor topology it works. In

问题 I create kafka streaming application using Processor API. Here is how i create a topic to attach timestamp to all incoming messages kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic topicName --config message.timestamp.type= CreateTime The workflow is processing the incoming messages from source topic and posting it to sink topic. For some strange reason, I have seen same timestamp coming in both source and sink topic messages. Say for ex, in

问题 H all, I have a requirement where in I need to re-ingest some of my older data. We have a multi staged pipeline , the source of which is a Kafka topic. Once a record is fed into that, it runs through a series of steps(about 10). Each step massages the original JSON object pushed to the source topic and pushes to a destination topic. Now, sometimes, we need to re ingest the older data and apply a subset of the steps I described above. We intend to push these re-ingest records to a different

问题 I am developing a Spark application that listens to a Kafka stream using Spark and Java. I use kafka_2.10-0.10.2.1. I have set various parameters for Kafka properties: bootstrap.servers , key.deserializer , value.deserializer , etc. My application compiles fine, but when I submit it, it fails with the following error: Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/kafka/common/serialization/StringDeserializer I do use StringDeserializer for key.deserializer and value

问题 Is it possible to use interactive query (InteractiveQueryService) within Spring Cloud Stream the class with @EnableBinding annotation or within the method with @StreamListener? I tried instantiating ReadOnlyKeyValueStore within provided KStreamMusicSampleApplication class and process method but its always null. My @StreamListener method is listening to a bunch of KTables and KStreams and during the process topology e.g filtering, I have to check whether the key from a KStream already exists

问题 I have a Kafka streams app to consume 1 source topic with 20 partitions. Traffic load is about 2K records/sec. I deployed the app to 63 instances and it's working fine. But I noticed that, the partition assignment is always changing. I checked the KafkaStreams#localTheadMetadata output for each instance, the response is always PARTITIONS_REVOKED or PARTITIONS_ASSIGNED , sometime it's RUNNING . From the log, I saw two different errors: Offset commit failed on partition production-smoke-KSTREAM

问题 What class/method in Kafka Streams can we use to serialize/deserialize Java object to byte array OR vice versa? The following link proposes the usage of ByteArrayOutputStream & ObjectOutputStream but they are not thread safe. Send Custom Java Objects to Kafka Topic There is another option to use the ObjectMapper, ObjectReader (for threadsafe), but that's converting from POJO -> JSON -> bytearray. Seems this option is an extensive one. Wanted to check if there is a direct way to translate

问题 I'm trying to make a practical design decision based on convention and plausibility with KafkaStreams. Let's say that I have two different events that I want to place into KTable s. I have a producer sending these messages to a KStream that is listening to that topic. From what I can tell I cannot use conditional forwarding for messages using KafkaStreams , so if the stream is subscribe to many topics (one for each of the above messages, for example) I can only call stream.to on a single sink

问题 So basically I have Account class. I have data. I want to send those objects into my topic with producer. That is okay for now. Later on, I want to do aggregation with Kafka Streams but I can not because some Serde properties is wrong in my configuration, I think :/. I dont know where the error is. My producer works fine, but I can't aggregate. Anyone help me to look my kafka streams code please? My Account class: public class Account { private long fromId; private long amount; private long

问题 I have a 2 Kafka Streams application. One application listens from say topic1 and produces to topic2 and the other listens from topic2 and produces to topic3 . The applications were working fine before the kafka broker went down. The broker came back up but the streams applications have stopped. Following is the exception of first streams app: Exception in thread "streams-collection-7cda47bc-a1db-4ad5-a3d4-bd8f8dc85bf4-StreamThread-1" org.apache.kafka.streams.errors.StreamsException: