apache-storm

问题 This is a general question regarding Unit Testing Bolts and Spouts in a Storm Topology written in Java. What is the recommended practice and guideline for unit-testing (JUnit?) Bolts and Spouts ? For instance, I could write a JUnit test for a Bolt , but without fully understanding the framework (like the lifecycle of a Bolt ) and the Serialization implications, easily make the mistake of Constructor-based creation of non-serializable member variables. In JUnit, this test would pass, but in a

问题 We are new to storm. We don't know about how to create topology please help us to work with storm. We tried sample wordcount c=topology given in the article "running storm on windows". But we can't understand how to give an input and also where the input is present and where the output is present in the storm ui. 回答1: Input and output does not present in Storm UI.In Storm UI you can see no of tuples emitted,processing time,cluster configuration and health of your cluster.To see output and

问题 We can generate storm jar using particular parameter. However, if we need to debug this project (actually a far) locally as well as remotely? If it is simple jar, that we can debug. However, here we are deploying jar using following command: storm jar project.jar main_class_name Not sure how can we deploy storm topology, so that we can do storm project in debugging mode? Please find updated yaml file as below: # Licensed to the Apache Software Foundation (ASF) under one # or more contributor

问题 im trying to setup a project , storm from git https://github.com/nathanmarz/storm/wiki/Setting-up-development-environment Download a Storm release , unpack it, and put the unpacked bin/ directory on your PATH My question is what is PATH mean, what exactly they want me to do ? Sometimes I see some /bin/path , $PATH, echo PATH can someone explain the concept of the PATH , so I can setup everything easily in the future without just blindly following the instructions? This is definitely techincal

问题 I saw many code snippets in which a loop was used inside Spout.nextTuple() (for example to read a whole file and emit a tuple for each line): public void nextTuple() { // do other stuff here // reader might be BufferedReader that is initialized in open() String str; while((str = reader.readLine()) != null) { _collector.emit(new Values(str)); } // do some more stuff here } This code seems to be straight forward, however, I was told that one should not loop inside nextTuple() . The question is

问题 When trying to submit my topology through StormSubmitter, I am getting - Caused by: java.lang.NoSuchFieldError: INSTANCE at org.apache.http.impl.io.DefaultHttpRequestWriterFactory.<init>(DefaultHttpRequestWriterFactory.java:52) I am using Spring. I am not initializing HttpClient in Spout/Bolt Constructor. Instead its initialized in constructor of a class that is being fetched from Spring Context in prepare() method of bolt Code is structured as follows - SomeBolt.java @Component public class

问题 I am working on a core Java project. I am writing an Apache Storm topology and need to exclude storm jars while binding the topology into jar. Is there any way to do this without using maven? I know with maven we can use <scope>provided</scope> but I need an alternative to this. PS: I am using Eclipse. 回答1: If you are using Maven instead of Gradle, and you come here for excluding lib of Storm in building, I have the solution and a working example. The documentation of Storm tells us to

问题 Now I am learning Storm's Guaranteeing Message Processing and am confused by some concepts in this part. To guarantee a message emitted by a spout is fully processed, Storm uses acker to achieve this. Each time a spout emits a tuple, acker will assign "ack val" initialized as 0 to store the status of the tuple tree. Each time the downstream bolts of this tuple emit new tuple or ack an "old" tuple, the tuple ID will be XOR with "ack val". The acker only needs to check whether "ack val" is 0 or

问题 H all, I have a requirement where in I need to re-ingest some of my older data. We have a multi staged pipeline , the source of which is a Kafka topic. Once a record is fed into that, it runs through a series of steps(about 10). Each step massages the original JSON object pushed to the source topic and pushes to a destination topic. Now, sometimes, we need to re ingest the older data and apply a subset of the steps I described above. We intend to push these re-ingest records to a different

问题 I'm getting this following error while executing a java code which is using Kafka into Storm. ERROR backtype.storm.util - Async loop died! java.nio.BufferUnderflowException: null 来源： https://stackoverflow.com/questions/26879748/error-backtype-storm-util-async-loop-died-java-nio-bufferunderflowexception