storm kafka first message is skipped during restart and first start as well

孤人 提交于 2019-12-11 14:21:45

问题


I have written a sample topology where, it will consume the message from kafka and log it. please find the code snippet below

End to End topology is fine. When I post the message in Kafka Producer it's consumed properly. I simply get the message and log it in MessagePrinter.

Issue described below

use case 1: I have brought down the topology, sent messages 1-10, when I bring up the topology, message 2-10 is logged properly by topology and first message alone is not logged.

use case 2: same issue is happening when the topology is started, first message is not processed and upcoming messages are logged properly by topology

public class Topology extends BaseTopologyBuilder { 
//overridden "apply" method where from the record only message is extracted out "record.value()"
private static Func<ConsumerRecord<String, String>, List<Object>> FUNCTION = new CommandValueFunction();
public void config() throws IOException {
    config = new Config();
    config.setDebug(false);
    config.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, 1)));
    config.setNumWorkers(1);
    config.put("topology.spout.max.batch.size", 1);
}
protected KafkaSpoutConfig<String, String> spoutConfig(String topic) {
    return KafkaSpoutConfig
            .builder(localhost:9092, topic)
            .setGroupId("kafkaSpoutTestGroup")
            .setMaxPartitionFectchBytes(2000000000)
            .setRecordTranslator(FUNCTION, new Fields("message"))
            .setRetry(newRetryService()).setOffsetCommitPeriodMs(10_000)
            .setFirstPollOffsetStrategy(FirstPollOffsetStrategy.UNCOMMITTED_LATEST)
            .setMaxUncommittedOffsets(250)
            .setProp(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true")
            .build();
}
//MessagePrinter extends BaseFunction and tuples are printed in execute()
public void buildTopology(MessagePrinter bolt) {
    TridentTopology topology = new TridentTopology();
    Stream stream = topology.newStream("spout", new KafkaTridentSpoutOpaque<>(spoutConfig("sample")));
    stream.each(new Fields("message"), bolt, new Fields()).parallelismHint(2);
    try {
        config.put("zookeeper.ip", localhost:2182);
        StormSubmitter.submitTopology("topology", config, topology.build());
    } catch (Exception exception) { }
}
public static void main(String[] args) throws Exception {
    Topology topologyBuilder = new Topology();
    topologyBuilder.config();
    topologyBuilder.buildTopology();
}}

来源:https://stackoverflow.com/questions/48419985/storm-kafka-first-message-is-skipped-during-restart-and-first-start-as-well

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!