apache-kafka-streams

I have a Kafka topic where I send location events (key=user_id, value=user_location). I am able to read and process it as a KStream : KStreamBuilder builder = new KStreamBuilder(); KStream<String, Location> locations = builder .stream("location_topic") .map((k, v) -> { // some processing here, omitted form clarity Location location = new Location(lat, lon); return new KeyValue<>(k, location); }); That works well, but I'd like to have a KTable with the last known position of each user. How could I do it? I am able to do it writing to and reading from an intermediate topic: // write to

State stores in Kafka Streams are created internally. State stores are partitioned by key, but do not allow to provide partitioning other than by key (to my knowledge). QUESTIONS How to control the number of partitions of a state-store internally created topic ? How does the state store topic infer the number of partitions and the partitioning to use by default, and how to override? How to work it around if you want to partition your state-store by something other than the key of your incoming key-value record and have co-partitioning? In this case, I'd like to partition by something more

My goal is to consume from topic A, do some processing and produce to topic B, as a single atomic action. To achieve this I see two options: Use a spring-kafka @Kafkalistener and a KafkaTemplate as described here . Use Streams eos (exactly-once) functionality. I have successfully verified option #1. By successfully, I mean that if my processing fails (IllegalArgumentException is thrown) the consumed message from topic A keeps being consumed by the KafkaListener. This is what I expect, as the offset is not committed and DefaultAfterRollbackProcessor is used. I am expecting to see the same

I would like to create a Kafka stream-based application that processes a topic and takes messages in batches of size X (i.e. 50) but if the stream has low flow, to give me whatever the stream has within Y seconds (i.e. 5). So, instead of processing messages one by one, I process a List[Record] where the size of the list is 50 (or maybe less). This is to make some I/O bound processing more efficient. I know that this can be implemented with the classic Kafka API but was looking for a stream-based implementation that can also handle offset committing natively, taking errors/failures into account

In Kafka Stream WordCount example, it uses StateStore to store word counts. If there are multiple instances in the same consumer group, the StateStore is global to the group, or just local to an consumer instance? Thnaks Matthias J. Sax This depends on your view on a state store. In Kafka Streams a state is shared and thus each instance holds part of the overall application state. For example, using DSL stateful operator use a local RocksDB instance to hold their shard of the state. Thus, with this regard the state is local. On the other hand, all changes to the state are written into a Kafka

I'm trying to use kafka streams to perform a windowed aggregation and emit the result only after a certain session window is closed. To achieve this I'm using the suppress function. The problem is that I don't find a way to make this simple test work because when it tries to persist the state I get a class cast exception because it tries to cast Windowed to String. I have tried to provide to the aggregate function a Materialized<Windowed<String>,Long,StateStore<>> but it doesn't type check because it expects the first type to be simply string. What am I missing here? kafka version 2.1.1