apache-storm

Apache Kafka: Distributed messaging system Apache Storm: Real Time Message Processing How we can use both technologies in a real-time data pipeline for processing event data? In terms of real time data pipeline both seems to me do the job identical. How can we use both the technologies on a data pipeline? You use Apache Kafka as a distributed and robust queue that can handle high volume data and enables you to pass messages from one end-point to another. Storm is not a queue. It is a system that has distributed real time processing abilities, meaning you can execute all kind of manipulations

I am trying to rebalance the number of executors for a bolt in a running Apache Storm (0.9.5) topology. When I execute the command to the Nimbus node ./storm rebalance MyTopology -n 2 -e GreenBolt=4 it accepts the command line input but the number of executors is not changed when I view in the Storm UI. Is there a limitation that I'm not aware of like a rebalance can not increase the total number of executors and only move them from one bolt to another? You always need to have more (or equal number of) tasks than executors. As the number of tasks is fixed, you need to set a larger initial

I found this statement in the help of a project that I want to import it named "storm- election" This is a simple demo app based on the storm-starter project. https://github.com/nathanmarz/storm-starter . So, I imported storm-starter project successfully. How can I import storm-election project? Can I import a project on another imported project ? You cannot create a project under any project, like you create folder(s) under folder, folder(s) under project, file(s) under project and file(s) under folder. All projects in a workspace are siblings no child and parent relationship exists between

I'm trying to run a topology in storm that makes calls to python (ex: WordCountTopology) but I encounter errors that are related to the fact that python3.5.2 is the default python on my server (errors are about the old/new syntax of print command). How to specify to storm to use python2.7 instead of python3.5? Setting a python alias to python2.7 does not change anything. Any help appreciated. I guess you're using ShellSpout / ShellBolt. In constructor you can specify the command to execute subprocess, so you can explicitly set command to python2.7. For example, public static class

Below is a screenshot of my topologies' Storm UI. This was taken after the topology finished processing 10k messages. (The topology is configured with 4 workers and uses a KafkaSpout). The sum of the "process latency" of my bolts is about 8100ms and the complete latency of the topology is a much longer 115881ms. I'm aware that these sort of discrepancies can occur due to resource contention or something related to Storm internals. I believe resource contention is not an issue here; the GC didn't run at all during this test and profiling shows that I have plenty of available CPU resources. So I

I'm running storm topology " pyleus --verbose local xyz_topology.jar " using storm-1.0.0, pyleus-0.3.0, centos-6.6 and getting the Error Exception in thread "main" java.lang.NoClassDefFoundError: backtype/storm/topology/IRichBolt Running: java -client -Ddaemon.name= -Dstorm.options= -Dstorm.home=/usr/local/apache-storm-1.0.0 -Dstorm.log.dir=/usr/local/apache-storm-1.0.0/logs -Djava.library.path=/usr/local/ lib:/opt/local/lib:/usr/lib -Dstorm.conf.file= -cp /usr/local/apache-storm-1.0.0/lib/asm-5.0.3.jar: /usr/local/apache-storm-1.0.0/lib/clojure-1.7.0.jar:/usr/local/apache-storm-1.0.0/lib

I was looking for how to kill a topology from a storm bolt in storm 1.2.1 in my scala project. From this answer , I wrote following code to do it: private def shutTopology(){ import org.apache.storm.utils.Utils import org.apache.storm.utils.NimbusClient val conf = Utils.readStormConfig val nimbusClient = NimbusClient.getConfiguredClient(conf).getClient nimbusClient.deactivate(topology_name) } But it given following error in line: val conf = Utils.readStormConfig , How to resolve this? Error:(46, 17) package daemon contains object and package with same name: nimbus one of them needs to be

I'm new to java and Storm so please forgive any obvious mistakes. I'm trying to run storm with a flume connector but It crashes with the following error: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297) at java.lang.Thread.run(Thread.java:744) Caused

I have a problem.I understad trouble but I don't find solution.Cluster isn't shutdown,throw IOException.I/O methods not work because Windows doesn't allow temp folder.I mean this path; C:\Users\Mert\AppData\Local\Temp\8bbdc8e5-a753-40f3-afe0-0c2fcc1c25a0\version-2\log.1 Storm be able to delete log files for shutdown. I tried administrator mode for Eclipse.exe and javaw.exe but it doesn't work.What can I do ? IO Exception stack trace: exception.detailMessage -> java.io.IOException: Unable to delete file: C:\Users\Mert\AppData\Local\Temp\c3c9b21b-7817-4b63-92ae-555a49743d9d\version-2\log.1