kafka-consumer-api

When a Kafka Consumer fails to deserialize a message, is it the client applications responsibility to deal with the Poison Message? Or Does Kafka "increment" the message offset and continue consumption of valid messages? Is there a "Best Practice" for dealing with Poison Messages held on Kafka topics? When Kafka is unable to deserialize the record the consumer will receive a org.apache.kafka.common.KafkaException , you should commit the offset yourself and keep consuming. 来源： https://stackoverflow.com/questions/35991849/how-does-should-kafka-consumer-cope-with-poison-messages

I'm trying to obtain and store the offset for a specific message in Kafka by using Spark Direct Stream. Looking at the Spark documentation is simple to obtain the range offsets for each partition but what I need is to store the start offset for each message of a topic after a full scan of the queue. Yes, you can use MessageAndMetadata version of createDirectStream which allows you to access message metadata . You can find example here which returns Dstream of tuple3 . val ssc = new StreamingContext(sparkConf, Seconds(10)) val kafkaParams = Map[String, String]("metadata.broker.list" ->

I have thousands of files generated each day which I want to stream using Kafka. When I try to read the file, each line is taken as a separate message. I would like to know how can I make each file's content as a single message in Kafka topic and with consumer how to write each message from Kafka topic in a separate file. Rambler You can write your own serializer/deserializer for handling files. For example : Producer Props : props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, org.apache.kafka.common.serialization.StringSerializer); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,

I am new in Kafka, I use kafka to collect netflow through logstash(it is ok), and I want to send the data to elasticsearch from kafka, but there are some problems. My question is how can I connect Kafka with Elasticsearch? netflow to kafka logstash config: input{ udp{ host => "120.127.XXX.XX" port => 5556 codec => netflow } } filter{ } output { kafka { bootstrap_servers => "localhost:9092" topic_id => "test" } stdout{codec=> rubydebug} } kafka to elasticsearch logstash: input { kafka { } } output { elasticsearch { hosts => ["120.127.XXX.XX:9200"] } stdout{codec=> rubydebug} } log: D:\ELK

I'm receiving exception when start Kafka consumer. org.apache.kafka.clients.consumer.OffsetOutOfRangeException: Offsets out of range with no configured reset policy for partitions{test-0=29898318} I'm using Kafka version 9.0.0 with Java 7. So you are trying to access offset( 29898318 ) in topic( test ) partition( 0 ) which is not available right now. There could be two cases for this Your topic partition 0 may not have those many messages Your message at offset 29898318 might have already deleted by retention period To avoid this you can do one of following: Set auto.offset.reset config to