apache-kafka-streams

问题 After listening on a kafka topic using @StreamListener, upon RuntimeException, global erroChannel or topic specific errorChannel (topic.group.errors) not receiving any error message. @ServiceActivator not receiving anything. POM Dependencies : Greenwich.RELEASE <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-stream</artifactId> </dependency> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-stream-schema</artifactId> </dependency>

问题 We're trying to achieve a deduplication service using Kafka Streams. The big picture is that it will use its rocksDB state store in order to check existing keys during process. Please correct me if I'm wrong, but to make those stateStores fault tolerant too, Kafka streams API will transparently copy the values in the stateStore inside a Kafka topic ( called the change Log). That way, if our service falls, another service will be able to rebuild its stateStore according to the changeLog found

问题 Suppose I have 2 types of logs, which have a common field 'uid', and I want to output the log if the log of both of these 2 logs containing the uid arrives, like a join, is it possible for Kafka ? 回答1: Yes, absolutely. Check out Kafka Streams, specifically the DSL API. It goes something like: StreamsBuilder builder = new StreamsBuilder(); KStream<byte[], Foo> fooStream = builder.stream("foo"); KStream<byte[], Bar> barStream = builder.stream("bar"); fooStream.join(barStream, (foo, bar) -> {

问题 I have two Java Application (App1, App2) to test how to access a KTable from a different app on a single instance environment in docker. The first App (App1) writes to a KTable with following code. public static void main(String[] args) { final Properties props = new Properties(); props.put(StreamsConfig.APPLICATION_ID_CONFIG,"gateway-service"); props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "172.18.0.11:9092"); props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG,Serdes.String()

问题 I have a Kafka topic called A . format of data in topic A is : { id : 1, name:stackoverflow, created_at:2017-09-28 22:30:00.000} { id : 2, name:confluent, created_at:2017-09-28 22:00:00.000} { id : 3, name:kafka, created_at:2017-09-28 24:42:00.000} { id : 4, name:apache, created_at:2017-09-28 24:41:00.000} Now in consumer side i want to get only latest data of one hour window means every one hour i need to get latest value from topic based on created_at My expected output is : { id : 1, name

问题 this is a followup question of : Kafka Streams - How to scale Kafka store generated changelog topics let's hypothetically assume the stream consumer needs to do some transformation before storing the data (indexing by v->k instead of k->v). At the end, the goal is that each consumer needs to store the full set of transformed record (v->k) in a rocksDB. I understand another processor upstream could take care of producing v->k based on k->v and the final consumer could simply materialized the

问题 I'm trying to implement a simple CQRS/event sourcing proof of concept on top of Kafka streams (as described in https://www.confluent.io/blog/event-sourcing-using-apache-kafka/) I have 4 basic parts: commands topic, which uses the aggregate ID as the key for sequential processing of commands per aggregate events topic, to which every change in aggregate state are published (again, key is the aggregate ID). This topic has a retention policy of "never delete" A KTable to reduce aggregate state

问题 I'm currently incapable of deserialize an avro PRIMITIVE key in a KSTREAM APP the key in encoded with an avro schema ( registered in the schema registry ) , when i use the kafka-avro-console-consumer, I can see that the key is correctly deserialize But impossible to make it work in a KSTREAM app the avro schema of the key is a PRIMITIVE: {"type":"string"} I already followed the documentation of confluent final Serde<V> valueSpecificAvroSerde = new SpecificAvroSerde<>(); final Map<String,

问题 I create a Kafka Streams application, which receives different JSON objects from different topics and I want to implement some kind of wait function, but I'm not sure about how to implement it best. To simplify the problem I'll use simplified entities in the following section, I hope the problem can be described very good with it. So in one of my streams I receive car objects and every car has an id. In a second stream I receive person objects and every person has also a car id and is

问题 I'm just starting into the topic of real-time stream data processing frameworks, and I have a question to which I as of yet could not find any conclusive answer: Do the usual suspects (Apache's Spark, Kafka, Storm, Flink etc.) support processing data with an event time resolution of nanoseconds (or even picoseconds)? Most people and documentation talk about a millisecond or microsecond resolution, but I was unable to find a definite answer if more resolution would be possible or a problem.