apache-kafka-streams

I am developing a simple Kafka Stream application which extracting messages from a topic and put it into another topic after transformation. I am using Intelij for my development. When I debug/run this application, it works perfect if my IDE and the Kafka Server sitting in the SAME machine (i.e. with the BOOTSTRAP_SERVERS_CONFIG = localhost:9092 and SCHEMA_REGISTRY_URL_CONFIG = localhost:8081) However, when I try to use another machine to do the development (i.e. with the BOOTSTRAP_SERVERS_CONFIG = XXX.XXX.XXX:9092 and SCHEMA_REGISTRY_URL_CONFIG = XXX.XXX.XXX:8081 where XXX.XXX.XXX is the ip

This question already has an answer here: How to send final kafka-streams aggregation result of a time windowed KTable? 2 answers I have a KStream in which I want to count some dimension of the events. I do it as follows: KTable<Windowed<Long>, Counter> ret = input.groupByKey() .windowedBy(TimeWindows.of(Duration.of(10, SECONDS))) .aggregate(Counter::new, (k, v, c) -> new Counter(c.count + v.getDimension())); I want to have a new KStream with those aggregations as events. I can do it easily like this: ret.toStream().to("output"); The problem is that every event in "input" topic will produce an

The naive approach for implementing the use case of enriching an incoming stream of events stored in Kafka with reference data - is by calling in map() operator an external service REST API that provides this reference data, for each incoming event. eventStream.map((key, event) -> /* query the external service here, then return the enriched event */) Another approach is to have second events stream with reference data and store it in KTable that will be a lightweight embedded "database" then join main event stream with it. KStream<String, Object> eventStream = builder.stream(..., "event-topic"

With the addition of Headers to the records ( ProducerRecord & ConsumerRecord ) in Kafka 0.11, is it possible to get these headers when processing a topic with Kafka Streams? When calling methods like map on a KStream it provides arguments of the key and the value of the record but no way I can see to access the headers . It would be nice if we could just map over the ConsumerRecord s. ex. KStreamBuilder kStreamBuilder = new KStreamBuilder(); KStream<String, String> stream = kStreamBuilder.stream("some-topic"); stream .map((key, value) -> ... ) // can I get access to headers in methods like

Based on apache Kafka docs KStream-to-KStream Joins are always windowed joins , my question is how can I control the size of the window? Is it the same size for keeping the data on the topic? Or for example, we can keep data for 1 month but join the stream just for past week? Is there any good example to show a windowed KStream-to-kStream windowed join? In my case let's say I have 2 KStream, kstream1 and kstream2 I want to be able to join 10 days of kstream1 to 30 days of kstream2 . That is absolutely possible. When you define you Stream operator, you specify the join window size explicitly.

I use Kafka streams to process real-time data, in the Kafka streams tasks, I need to access MySQL to query data, and need to call another restful service. All the operations are synchronous. I'm afraid the sync call will reduce the process capability of the streams tasks. Is this a good practice? or Is there any good idea to do this? A better way to do it would be to stream your MySQL table(s) into Kafka, and access the data there. This has the advantage of decoupling your streams app from the MySQL database. If you moved away from MySQL in the future, so long as the data were still written to

I have a Kafka topic where I expect messages with two different key types: old and new. i.e. "1-new" , "1-old" , "2-new" , "2-old" . Keys are unique, but some might be missing. Now using Kotlin and KafkaStreams API I can log those messages with have same key id from new and old. val windows = JoinWindows.of(Duration.of(2, MINUTES).toMillis()) val newStream = stream.filter({ key, _ -> isNew(key) }) .map({key, value -> KeyValue(key.replace(NEW_PREFIX, ""), value) }) val oldStream = stream.filter({ key, _ -> isOld(key) }) .map({key, value -> KeyValue(key.replace(OLD_PREFIX, ""), value) }) val