apache-storm

We need to pass an object when creating a topology so that the bolt can access that and do some further processing based on that object. Is it possible to pass the object via TopplogyContext and if yes, how? Or are there any other ways to pass an object when submitting a topology, before submitting so that bolt can have a handle/control on it? We need to pass the object via a context so that all bolts can access it and there is no need to force an implementation of constructor in all the bolts for that topology. So, Wanted to know if any API exists to do the same? You can pass the object in

I am trying to share the task among the multiple spouts. I have a situation, where I'm getting one tuple/message at a time from external source and I want to have multiple instances of a spout, main intention behind is to share the load and increase performance efficiency. I can do the same with one Spout itself, but I want to share the load across multiple spouts. I am not able to get the logic to spread the load. Since the offset of messages will not be known until the particular spout finishes the consuming the part (i.e based on buffer size set). Can anyone please put some bright light on

I want to turn off the logging feature offered by default when we run from local cluster. Currently its logging so many information on the console. Below is the example of log: 261 [main] INFO backtype.storm.daemon.task - Shut down task Getting-Started-Toplogie-1-1376388324:2 2261 [main] INFO backtype.storm.daemon.task - Shutting down task Getting-Started-Toplogie-1-1376388324:1 2262 [Thread-24] INFO backtype.storm.util - Async loop interrupted! 2276 [main] INFO backtype.storm.daemon.task - Shut down task Getting-Started-Toplogie-1-1376388324:1 2278 [main] INFO backtype.storm.daemon.worker -

I followed Storm Starter instructions and imported Twitter Storm in IntelliJ. For sake of testing I edited ExclaimationToplogy a bit and used following Maven command to build and run it: mvn -f m2-pom.xml compile exec:java -Dstorm.topology=storm.starter.ExclamationTopology But I am more interested to build and run within IDE rather than from command line. What actions do I need to perform? Thanks Lukas Hajdu Follow the steps in storm-starter's: Using storm-starter with IntelliJ IDEA Open Maven's pom.xml file and remove <scope>provided</scope> line from storm dependency. This enables IntelliJ

Just want to make sure I got how Ack-ing works in Storm. I have 1 spout and 2 bolts chained together. Spout emits tuple to Bolt1 which in turn will emit a tuple to Bolt 2. I want Bolt 2 to ack the initial tuple sent from Spout and I'm not sure how. In order to guarantee fault tolerance (ie: tuples are resent) I want to ack in bolt 2 the tuple emitted by Spout just in case it fails somewhere in the process so it can be resent. Consider this example: Spout: _collector.emit(new Values(queue.dequeue()) Bolt1: def execute(tuple: Tuple) { _collector.emit(tuple, new Values("stuff")) } At this point

This is a question regarding how storm's max spout pending works. I currently have a spout that reads a file and emits a tuple for each line in the file (I know storm is not the best solution for dealing with files but I do not have a choice for this problem). I set the max spout pending to 50k to throttle how many tuples get into the topology to be processed. However, I see this number not having any effect in the topology. I see all records in a file being emitted every time. My guess is this might be due to a loop I have in the nextTuple method that emits all records in a file. So my

Is there a clean way to stop storm without killing it with "kill XXX" where XXX is the PID? I run "storm kill topology-name" to kill the topology, but after that, is there a clean way to shutdown workers, nimbus, supervisor and ui? I didn't find any command which corresponds to this from the documentation: https://github.com/nathanmarz/storm/wiki/Command-line-client command to kill the topology ( as you rightly mentioned ) storm kill topology-name Now to shutdown the cluster you need to run the following command to each node sudo service supervisord stop Give supervisord a few seconds to

I have a prototype storm app that reads a STOMP stream and stores the output on HBase. It works, but is not very flexible and I'm trying to get it set up in a more consistent way with the rest of our apps, but not having much luck figuring out how the current way of working with Storm. We use spring-jms classes, but instead of using them in the standard spring way, they are being created at run time, and setting dependencies manually. This project: https://github.com/granthenke/storm-spring looked promising, but it hasn't been touched in a couple years and doesn't build properly since the