Kafka SessionWindow with suppress only sends final event when there is a steady stream of input records

Question

It appears as though a Kafka Stream with a session window with a grace period and suppression fails to output a final event if there is no constant stream of input records.<

Answer 1

Update : I have an error in topology, fixed

I had the same exactly same problems as you when using suppress, and it's expected behavior. Because suppress only support emit buffered records using stream time not wall-clock time, if you stop getting new records, stream time will be freeze and Suppress will not emit the last suppressed window.

The solution I used is to write a custom suppress using Processor API (use a Transfomer so you can use DSL to send supprssed record downstream) with a state store used as a buffer, then check what windows should be flush (or emit) to downstream processor whenever there is a new record come in or after a time interval has passed (using a WALL_CLOCK_TIME punctuate).

The transfomer would look like this:

public class SuppressWindowTransformer implements Transformer<Windowed<String>, String, KeyValue<Windowed<String>, String>> {
    private ProcessorContext context;
    private Cancellable cancellable;
    private KeyValueStore<Windowed<String>, String> kvStore;
    @Override
    public void init(ProcessorContext context) {
        this.context = context;
        kvStore = (KeyValueStore) context.getStateStore("suppressed_windowed_transaction");
        cancellable = context.schedule(Duration.ofMillis(100), PunctuationType.WALL_CLOCK_TIME, timestamp -> flushOldWindow());
    }

    @Override
    public KeyValue<Windowed<String>, String> transform(Windowed<String> key, String value) {
        kvStore.put(key, value);//buffer (or suppress) the new in coming window
        flushOldWindow();
        return null;
    }

    private void flushOldWindow() {
        //your logic to check for old windows in kvStore then flush

        //forward (or unsuppressed) your suppressed records downstream using ProcessorContext.forward(key, value)
    }

    @Override
    public void close() {
        cancellable.cancel();//cancel punctuate
    }
}

And in your Stream DSL:

stream.groupByKey()
            .windowedBy(SessionWindows.with(Duration.ofMillis(400)).grace(Duration.ofMillis(0)))
            .aggregate(...)//remove suppress operator and write custom suppress using processor API
            .toStream()
            .transform(SuppressWindowTransformer::new, "suppressed_windowed_transaction")
            .to("throughput-test-aggregated");