apache-kafka-streams

I have a Kafka producer which is producing messages at high rate (message key is let us say a username and value is his current score in a game). The Kafka consumer is relatively slow in processing the consumed messages. Here my requirement is to show most up-to-date score and avoid showing stale data, with the tradeoff that some scores may never be shown. Essentially for each of the username, I may have hundreds of messages in the same partition, but I always want to read the latest one. A crude solution which has been implemented was like this: The producer sends just a key as each message

I need to create a state store with String key HashMap as value. I tried the below two methods. // First method StateStoreSupplier avgStoreNew = Stores.create("AvgsNew") .withKeys(Serdes.String()) .withValues(HashMap.class) .persistent() .build(); // Second method HashMap<String ,Double> h = new HashMap<String ,Double>(); StateStoreSupplier avgStore1 = Stores.create("Avgs") .withKeys(Serdes.String()) .withValues(Serdes.serdeFrom(h.getClass())) .persistent() .build(); The code compiles fine without any error, but I get a runtime error io.confluent.examples.streams.WordCountProcessorAPIException

I'm writing an application with Kafka Streams (v0.10.0.1) and would like to enrich the records I'm processing with lookup data. This data (timestamped file) is written into a HDFS directory on daily basis (or 2-3 times a day). How can I load this in the Kafka Streams application and join to the actual KStream ? What would be the best practice to reread the data from HDFS when a new file arrives there? Or would it be better switching to Kafka Connect and write the RDBMS table content to a Kafka topic which can be consumed by all the Kafka Streams application instances? Update : As suggested

I am working with Kafka Stream 2.1 I am trying to write some test for a stream application that aggregates some events by their key (i.e by a correlation ID) using a session window with an inactivity gap of 300ms. Here is the aggregation implementation represented by a method : private static final int INACTIVITY_GAP = 300; public KStream<String, AggregatedCustomObject> aggregate(KStream<String, CustomObject> source) { return source // group by key (i.e by correlation ID) .groupByKey(Grouped.with(Serdes.String(), new CustomSerde())) // Define a session window with an inactivity gap of 300 ms

I've been successful in getting a streams app to consume, transform and produce data, but I've noticed that periodically, the streams processor will transition to a state of ERROR and the process will sit there without exiting. Showing me logs like: All stream threads have died. The instance will be in error state and should be closed. Is there a way to tell the Streams app to exit once its reached the ERROR state? Maybe a monitor thread of sorts? I see references in the comments of the Kafka Streams code to the user needing to close the application once its reached this state, however, I

I have an application that needs to listen to multiple different topics; each topic has separate logic for how the messages are handled. I had thought to use the same kafka properties for each KafkaStreams instance, but I get an error like the one below. Error java.lang.IllegalArgumentException: Assigned partition my-topic-1 for non-subscribed topic regex pattern; subscription pattern is my-other-topic Code (kotlin) class KafkaSetup() { companion object { private val LOG = LoggerFactory.getLogger(this::class.java) } fun getProperties(): Properties { val properties = Properties() properties.put