apache-storm-topology

问题 We have been struggling with this issue for a long time now. In short, our storm topology stops emitting messages from spout after some time in a random fashion. We have an automated script which re-deploys the topology at 06:00 UTC everyday after the master data refresh activity is complete. In the last 2 weeks, our topology stopped emitting the messages for 3 times in late UTC hours (between 22:00 and 02:00). It only comes online when we restart it which is around 06:00 UTC. I've searched

问题 Our Apache Storm topology listens messages from Kafka using KafkaSpout and after doing lot of mapping/reducing/enrichment/aggregation etc. etc finally inserts data into Cassandra. There is another kafka input where we receive user queries for data if topology finds a response then it sends that onto a third kafka topic. Now we want to write E2E test using Junit in which we can directly programmatically insert data into topology and then by inserting user query message, we can assert on third

问题 I am able to run storm Kafka with local cluster but not able to run with storm Submitter below is my topology code can anyone please help me to solve this issue :) package com.org.kafka; import org.apache.storm.Config; import org.apache.storm.LocalCluster; import org.apache.storm.generated.AlreadyAliveException; import org.apache.storm.generated.AuthorizationException; import org.apache.storm.generated.InvalidTopologyException; import org.apache.storm.kafka.KafkaSpout; import org.apache.storm

问题 We have a storm topology which interacts with a MariaDB database. Our Bolts implement the IRichBolt interface and override the lifecycle methods. We open a db connection in our prepare method and close it in the cleanup method. The cleanup method documentation says: Called when an IBolt is going to be shutdown. There is no guarentee that cleanup will be called, because the supervisor kill -9's worker processes on the cluster. The one context where cleanup is guaranteed to be called is when a

问题 I try to run a storm topology on a Storm cluster The topology jar is created by provided. Inside topology bolt i want to read data from MyIgniteCache module but i get such a following error. I think the dependencies (:/usr/hdp/2.6.0.3-8/storm/lib/log4j-slf4j-impl-2.8.jar:/usr/hdp/2.6.0.3-8/storm/lib/log4j-core-2.8.jar:log4j) of storm-core cause conflict over ignite-log4j. ava.lang.IncompatibleClassChangeError: Implementing class at java.lang.ClassLoader.defineClass1(Native Method) ~[?:1.8.0

问题 Our Apache Storm topology listens messages from Kafka using KafkaSpout and after doing lot of mapping/reducing/enrichment/aggregation etc. etc finally inserts data into Cassandra. There is another kafka input where we receive user queries for data if topology finds a response then it sends that onto a third kafka topic. Now we want to write E2E test using Junit in which we can directly programmatically insert data into topology and then by inserting user query message, we can assert on third

问题 I read a json from Kafka, FieldExtractionBolt reads that json extracts data into tuple values and passes them to CassandraWriterBolt, which in its turn writes a record in Cassandra writing all those tuple values into separate columns. JSON message on Kafka - {"pair":"GBPJPY","bid":134.4563,"ask":134.4354} FieldExtractionBolt - String message = tuple.getStringByField("message"); Map values = new Gson().fromJson(message, Map.class); basicOutputCollector.emit(new Values(values.get("pair"),

问题 I have written a sample topology where, it will consume the message from kafka and log it. please find the code snippet below End to End topology is fine. When I post the message in Kafka Producer it's consumed properly. I simply get the message and log it in MessagePrinter. Issue described below use case 1: I have brought down the topology, sent messages 1-10, when I bring up the topology, message 2-10 is logged properly by topology and first message alone is not logged. use case 2: same

问题 I am building a Java Spring application using Storm 1.1.2 and Kafka 0.11 to be launched in a Docker container. Everything in my topology works as planned but under a high load from Kafka, the Kafka lag increases more and more over time. My KafkaSpoutConfig: KafkaSpoutConfig<String,String> spoutConf = KafkaSpoutConfig.builder("kafkaContainerName:9092", "myTopic") .setProp(ConsumerConfig.GROUP_ID_CONFIG, "myGroup") .setProp(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, MyObjectDeserializer