End-of-window outer join with KafkaStreams

Question

I have a Kafka topic where I expect messages with two different key types: old and new. i.e. \"1-new\", \"1-old\", \"2-new\", \"

Answer 1

The DSL might not give you what you want. However, you can use Processor API. Having say this, the leftJoin can actually be used to do the "heavy lifting". Thus, after the leftJoin you can use .transform(...) with an attached state to "clean up" the data further.

For each old&null record you receive, put it into the store. If you receive a later old&new you can remove it from the store. Furthermore, you register a punctuation and on each punctuation call, you scan the store for entries that are "old enough" so you are sure no later old&new join result will be produced. For those entries, you emit old&null and remove from them from the store.

As an alternative, you can also omit the join, and do everything in a single transform() with state. For this, you would need to KStream#merge() old and new stream and call transform() on the merged stream.

Note: instead of registering a punctuation, you can also put the "scan logic" into the transform and execute it each time you process a record.