问题
Hi I'm having a issue with running storm cluster. It is similar to
My Topology is defined as :
package com.abc.newsclassification;
import StormBase.KnowledgeGraph.ClassifierBolt;
import StormBase.KnowledgeGraph.ClientSpecificTwitterSpout;
import StormBase.KnowledgeGraph.LiveTwitterSpout;
import StormBase.KnowledgeGraph.NewsTwitterSpout;
import StormBase.KnowledgeGraph.TwitterTrainingBolt;
import StormBase.KnowledgeGraph.UrlExtractorBolt;
import backtype.storm.Config;
import backtype.storm.LocalCluster;
import backtype.storm.StormSubmitter;
import backtype.storm.generated.AlreadyAliveException;
import backtype.storm.generated.InvalidTopologyException;
import backtype.storm.topology.TopologyBuilder;
import backtype.storm.tuple.Fields;
public class ClassifierTopology {
public static void main(String[] args) throws Exception {
TopologyBuilder builder = new TopologyBuilder();
// add a spout
builder.setSpout("spout", new NewsTwitterSpout(), 1);
// configure
Config conf = new Config();
conf.setDebug(false);
// submit it to the cluster, or submit it locally
conf.setMaxTaskParallelism(10);
LocalCluster cluster = new LocalCluster();
System.out.println(conf.entrySet());
cluster.submitTopology("testTopology", conf,
builder.createTopology());
}
}
------------------------------------------------------------------------------------------------------
** END OF FILE **
------------------------------------------------------------------------------------------------------
pom.xml:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>StormBase</groupId>
<artifactId>KnowledgeGraph</artifactId>
<version>0.0.1-SNAPSHOT</version>
<packaging>jar</packaging>
<name>KnowledgeGraph</name>
<url>http://maven.apache.org</url>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<repositories>
<repository>
<id>clojars.org</id>
<url>http://clojars.org/repo</url>
</repository>
</repositories>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>com.abc.newsclassification.ClassifierTopology</mainClass>
</manifest>
</archive>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id> <!-- this is used for inheritance merges -->
<phase>package</phase> <!-- bind to the packaging phase -->
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-jdk14</artifactId>
<version>1.6.1</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-nop</artifactId>
<version>1.6.0</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>storm</groupId>
<artifactId>storm</artifactId>
<version>0.8.2</version>
</dependency>
<dependency>
<groupId>org.apache.zookeeper</groupId>
<artifactId>zookeeper</artifactId>
<version>3.3.3</version>
<exclusions>
<exclusion>
<groupId>com.sun.jmx</groupId>
<artifactId>jmxri</artifactId>
</exclusion>
<exclusion>
<groupId>com.sun.jdmk</groupId>
<artifactId>jmxtools</artifactId>
</exclusion>
<exclusion>
<groupId>javax.jms</groupId>
<artifactId>jms</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.yammer.metrics</groupId>
<artifactId>metrics-core</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>com.101tec</groupId>
<artifactId>zkclient</artifactId>
<version>0.3</version>
</dependency>
<dependency>
<groupId>net.sf.jopt-simple</groupId>
<artifactId>jopt-simple</artifactId>
<version>4.5</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>1.9.2</version>
</dependency>
<dependency>
<groupId>com.netflix.curator</groupId>
<artifactId>curator-test</artifactId>
<version>1.2.5</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
<exclusion>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.twitter4j</groupId>
<artifactId>twitter4j-stream</artifactId>
<version>3.0.5</version>
</dependency>
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.2.1</version>
<type>jar</type>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.languagetool</groupId>
<artifactId>language-en</artifactId>
<version>2.3.1</version>
</dependency>
<dependency>
<groupId>org.twitter4j</groupId>
<artifactId>twitter4j-core</artifactId>
<version>3.0.5</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-core</artifactId>
<version>4.6.0</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-analyzers</artifactId>
<version>3.6.2</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-snowball</artifactId>
<version>3.0.3</version>
</dependency>
<dependency>
<groupId>org.apache.lucene</groupId>
<artifactId>lucene-analyzers-common</artifactId>
<version>4.6.0</version>
</dependency>
<dependency>
<groupId>com.gravity</groupId>
<artifactId>goose</artifactId>
<version>2.1.23</version>
</dependency>
<dependency>
<groupId>nz.ac.waikato.cms.weka</groupId>
<artifactId>weka-dev</artifactId>
<version>3.7.9</version>
</dependency>
<dependency>
<groupId>org.apache.directory.studio</groupId>
<artifactId>org.apache.commons.io</artifactId>
<version>2.4</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.1</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.0.8</version>
</dependency>
<dependency>
<groupId>org.javatuples</groupId>
<artifactId>javatuples</artifactId>
<version>1.2</version>
</dependency>
<dependency>
<groupId>org.apache.opennlp</groupId>
<artifactId>opennlp-tools</artifactId>
<version>1.5.3</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-io</artifactId>
<version>1.3.2</version>
</dependency>
<dependency>
<groupId>org.apache.directory.studio</groupId>
<artifactId>org.apache.commons.lang</artifactId>
<version>2.6</version>
</dependency>
<dependency>
<groupId>edu.stanford.nlp</groupId>
<artifactId>stanford-corenlp</artifactId>
<version>3.3.0</version>
</dependency>
<dependency>
<groupId>edu.stanford.nlp</groupId>
<artifactId>stanford-corenlp</artifactId>
<version>3.3.0</version>
<classifier>models</classifier>
</dependency>
<dependency>
<groupId>edu.washington.cs.knowitall.stanford-corenlp</groupId>
<artifactId>stanford-postag-models</artifactId>
<version>1.3.5</version>
</dependency>
<dependency>
<groupId>com.google.code.google-collections</groupId>
<artifactId>google-collect</artifactId>
<version>snapshot-20071022</version>
</dependency>
<dependency>
<groupId>com.googlecode.concurrent-trees</groupId>
<artifactId>concurrent-trees</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.16</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>1.0.3</version>
</dependency>
<dependency>
<groupId>org.hectorclient</groupId>
<artifactId>hector-core</artifactId>
<version>1.1-0</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>me.prettyprint</groupId>
<artifactId>hector-test</artifactId>
<version>1.0-5</version>
</dependency>
<dependency>
<groupId>FBKeyWords</groupId>
<artifactId>FBKeyWords</artifactId>
<version>0.0.1-SNAPSHOT</version>
<classifier>jar-with-dependencies</classifier>
</dependency>
<dependency>
<groupId>KeyWordExtractor</groupId>
<artifactId>KeyWordExtractor</artifactId>
<version>0.0.1-SNAPSHOT</version>
<classifier>jar-with-dependencies</classifier>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.6</version>
</dependency>
</dependencies>
</project>
---------------------------------------------------------------
** END OF pom.xml **
---------------------------------------------------------------
The following is the error I'm getting:
-----------------------------------------------------------------------------------------
[topology.workers=3, topology.debug=false]
Exception in thread "main" java.lang.RuntimeException: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:21)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:70)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:41)
at com.tookitaki.newsclassification.ClassifierTopology.main(ClassifierTopology.java:92)
Caused by: org.apache.thrift7.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.thrift7.transport.TSocket.open(TSocket.java:183)
at org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
at backtype.storm.security.auth.SimpleTransportPlugin.connect(SimpleTransportPlugin.java:66)
at backtype.storm.security.auth.ThriftClient.<init>(ThriftClient.java:46)
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:30)
at backtype.storm.utils.NimbusClient.<init>(NimbusClient.java:26)
at backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:19)
... 3 more
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:579)
at org.apache.thrift7.transport.TSocket.open(TSocket.java:178)
... 9 more
Please let me know which version of storm/other packages should I use, or if there are any other problem causing this Error. Any sugesstion is welcome.
回答1:
I also meet the same problem.
I check the iptables and supervisors and nimbus, but it isn't their problem.
I tried to submit half of the topology (by removing some kind of bolts) and it passed.
I check the nimbus.log and find a error line indicating that the thrift buffer is full , so you can change your "storm.yaml" with config:
nimbus.thrift.max_buffer_size: 20480000
回答2:
It looks like your topology wasn't submitted, because the storm client couldn't connect to Nimbus, in which case it wouldn't be anything in your code. Make sure nimbus is running and the storm command can access it, i.e. to do a "storm list".
回答3:
Old question but worth answering.....I am using Storm 1.0.1.
You should check if incoming port 6627 is open on Nimbus/Master from Storm Workers.
I got the same error when Storm workers was not able to connect to Nimbus and showing error on source download failure error from master.
Storm Master or Nimbus runs Thrift on a port 6627
You can check using command "netstat -nat" on your master/nimbus
tcp 0 0 0.0.0.0:6627 0.0.0.0:* LISTEN
回答4:
I encountered same problem, in my case problem is related about to zookeeper location settings.
I moved zookeeper nodes different servers and i forgot the change zookeeper locations in storm.yaml so nimbus can not get information from topologies and other storm-supervisor nodes.
After correction of wrong location of zookeeper nodes, the problem is solved.
As i understand your problem is solved but i want to mentioned my case for other readers.
来源：https://stackoverflow.com/questions/22264893/storm-topology-failure-while-running-on-production