问题
I have written a sample topology where, it will consume the message from kafka and log it. please find the code snippet below
End to End topology is fine. When I post the message in Kafka Producer it's consumed properly. I simply get the message and log it in MessagePrinter.
Issue described below
use case 1: I have brought down the topology, sent messages 1-10, when I bring up the topology, message 2-10 is logged properly by topology and first message alone is not logged.
use case 2: same issue is happening when the topology is started, first message is not processed and upcoming messages are logged properly by topology
public class Topology extends BaseTopologyBuilder {
//overridden "apply" method where from the record only message is extracted out "record.value()"
private static Func<ConsumerRecord<String, String>, List<Object>> FUNCTION = new CommandValueFunction();
public void config() throws IOException {
config = new Config();
config.setDebug(false);
config.put(Config.TOPOLOGY_MAX_SPOUT_PENDING, 1)));
config.setNumWorkers(1);
config.put("topology.spout.max.batch.size", 1);
}
protected KafkaSpoutConfig<String, String> spoutConfig(String topic) {
return KafkaSpoutConfig
.builder(localhost:9092, topic)
.setGroupId("kafkaSpoutTestGroup")
.setMaxPartitionFectchBytes(2000000000)
.setRecordTranslator(FUNCTION, new Fields("message"))
.setRetry(newRetryService()).setOffsetCommitPeriodMs(10_000)
.setFirstPollOffsetStrategy(FirstPollOffsetStrategy.UNCOMMITTED_LATEST)
.setMaxUncommittedOffsets(250)
.setProp(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true")
.build();
}
//MessagePrinter extends BaseFunction and tuples are printed in execute()
public void buildTopology(MessagePrinter bolt) {
TridentTopology topology = new TridentTopology();
Stream stream = topology.newStream("spout", new KafkaTridentSpoutOpaque<>(spoutConfig("sample")));
stream.each(new Fields("message"), bolt, new Fields()).parallelismHint(2);
try {
config.put("zookeeper.ip", localhost:2182);
StormSubmitter.submitTopology("topology", config, topology.build());
} catch (Exception exception) { }
}
public static void main(String[] args) throws Exception {
Topology topologyBuilder = new Topology();
topologyBuilder.config();
topologyBuilder.buildTopology();
}}
来源:https://stackoverflow.com/questions/48419985/storm-kafka-first-message-is-skipped-during-restart-and-first-start-as-well