apache-kafka-streams

I want to batch messages with KStream interface. I have a Stream with Keys/values I tried to collect them in a tumbling window and then I wanted to process the complete window at once. builder.stream(longSerde, updateEventSerde, CONSUME_TOPIC) .aggregateByKey( HashMap::new, (aggKey, value, aggregate) -> { aggregate.put(value.getUuid, value); return aggregate; }, TimeWindows.of("intentWindow", 100), longSerde, mapSerde) .foreach((wk, values) -> { The thing is foreach gets called on each update to the KTable. I would like to process the whole window once it is complete. As in collect Data from

For one of my Kafka streams apps, I need to use the features of both DSL and Processor API. My streaming app flow is source -> selectKey -> filter -> aggregate (on a window) -> sink After aggregation I need to send a SINGLE aggregated message to the sink. So I define my topology as below KStreamBuilder builder = new KStreamBuilder(); KStream<String, String> source = builder.stream(source_stream); source.selectKey(new MyKeyValueMapper()) .filterNot((k,v) -> k.equals("UnknownGroup")) .process(() -> new MyProcessor()); I define a custom StateStore and register it with my processor as below public

Had gone through multiple posts but most of them are related handling Bad messages not about exception handling while processing them. I want to know to how to handle the messages that is been received by the stream application and there is an exception while processing the message? The exception could be because of multiple reasons like Network failure, RuntimeException etc., Could someone suggest what is the right way to do? Should I use setUncaughtExceptionHandler ? or is there a better way? How to handle retries? Thanks in advance!! it depends what do you want to do with exceptions on

So I was evaluating the Kafka Streams and what it can do to see if it can fit my use case as I needed to do the aggregation of sensor's data for each 15min, Hourly, Daily and found it useful due to its Windowing feature. As I can create windows by applying windowedBy() on KGroupedStream but the problem is that windows are created in UTC and i want my data to be grouped by its originating timezone not by UTC Timezone as it hampers the aggregation so can any one help me on this. You can "shift" the timestamps using a custom TimestampExtractor -- before you write the result back into the output