apache-storm

My problem is that Storm KafkaSpout stopped to consume messages from Kafka topic after a period of time. When debug is enabled in storm, I get the log file like this: 2016-07-05 03:58:26.097 o.a.s.d.task [INFO] Emitting: packet_spout __metrics [#object[org.apache.storm.metric.api.IMetricsConsumer$TaskInfo 0x2c35b34f "org.apache.storm.metric.api.IMetricsConsumer$TaskInfo@2c35b34f"] [#object[org.apache.storm.metric.api.IMetricsConsumer$DataPoint 0x798f1e35 "[__ack-count = {default=0}]"] #object[org.apache.storm.metric.api.IMetricsConsumer$DataPoint 0x230867ec "[__sendqueue = {sojourn_time_ms=0.0

I was using kafka-storm to connect kafka and storm. I have 3 servers running zookeeper, kafka and storm. There is a topic 'test' in kafka that has 9 partitions. In the storm topology, the number of KafkaSpout executor is 9 and by default, the number of tasks should be 9 as well. And the 'extract' bolt is the only bolt connected to KafkaSpout, the 'log' spout. From the UI, there is a huge rate of failure in the spout. However, he number of executed message in bolt = the number of emitted message - the number of failed mesage in bolt. This equation is almost matched when the failed message is

I am new in storm and trying to understand the flow of execution of different methods from spout to bolt . Like spout has different methods like nextTuple() open() declareOutputFields() activate() deactivate() and bolt has methods like prepare() execute() cleanup() declareOutputFields() so can anyone tell me the sequence of execution of these methods ? First, when your topology is started... Create Spouts and Bolts declareOutputFields Spouts/Bolts serialized and assigned to workers Second, in each worker somewhere on the cluster... Spouts open and Bolts prepare (happens once) In a loop...

I want to know if it is possible to run kafka as a cloud-native application, and can I create a kafka cluster as a service on Pivotal Web Services. I don't want only client integration, I want to run the kafka cluster/service itself? Thanks, Anil I can point you at a few starting points, there would be some work involved to go from those starting points to something fully functional. One option is to deploy the kafka cluster on Cloud Foundry (e.g. Pivotal Web Services) using docker images. Spotify has Dockerized kafka and kafka-proxy (including Zookeeper). One thing to keep in mind is that PWS

I have a bolt that is making an API call (HTTP Get) for every tuple. to avoid the need to wait for the response, I was looking to use the apache HttpAsyncClient. after instantiating the client in the bolt's prepare method, the execute method constructs the URL from the tuple and calls sendAsyncGetRequest(url): private void sendAsyncGetRequest(String url){ httpclient.execute(new HttpGet(url), new FutureCallback<HttpResponse>() { @Override public void completed(HttpResponse response) { LOG.info("Response Code : " + response.getStatusLine()); LOG.debug(response.toString()); } @Override public

Let's say that I have 2 spouts and 3 bolts in Storm cluster and there are two worker nodes. Will be these spouts and bolts shared among these workers (for example first worker has 1 spout and 2 bolts, the second has 1 spout and 1 bolt) or each worker has 2 spouts and 3 bolts which ends up with 4 spouts and 6 bolts in whole cluster? Spout and bolt are shared by all your cluster (so worker). If you have 2 spouts and 3 bolts for 2 workers, they will be balanced between your 2 workers. You can use the ui (./nimbus ui) to visualise that :). In storm, a supervisor has multiple worker(processes)

Compile no problem, but after run..... 26183 [Thread-34] ERROR backtype.storm.util - Async loop died! java.lang.UnsatisfiedLinkError: org.zeromq.ZMQ$Socket.finalize()V at org.zeromq.ZMQ$Socket.finalize(Native Method) at org.zeromq.ZMQ$Socket.close(ZMQ.java:339) at storm.starter.spout.RandomSentenceSpout.nextTuple(RandomSentenceSpout.java:56) at backtype.storm.daemon.executor$fn__3985$fn__3997$fn__4026.invoke(executor.clj:502) at backtype.storm.util$async_loop$fn__465.invoke(util.clj:377) at clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:724) 26185 [Thread-34] ERROR