Could not parse Master URL: 'spark:http://localhost:18080'

问题:

When I'm trying to run my code it throws this Exception:

Exception in thread "main" org.apache.spark.SparkException: Could not parse Master URL:spark:http://localhost:18080 

This is my code:

SparkConf conf = new SparkConf().setAppName("App_Name").setMaster("spark:http://localhost:18080").set("spark.ui.port","18080"); JavaStreamingContext ssc = new JavaStreamingContext(sc, new Duration(1000)); String[] filet=new String[]{"Obama","ISI"};  JavaReceiverInputDStream<Status> reciverStream=TwitterUtils.createStream(ssc,filet); JavaDStream<String> statuses = reciverStream.map(new Function<Status, String>() {      public String call(Status status) { return status.getText(); }      }       ); ssc.start(); ssc.awaitTermination();}} 

Any idea how can I fix this problem?

回答1:

The problem is that you specify 2 schemas in the URL you pass to SparkConf.setMaster().

The spark is the schema, so you don't need to add http after spark. See the javadoc of SparkConf.setMaster() for more examples.

So the master URL you should be using is "spark://localhost:18080". Change this line:

SparkConf conf = new SparkConf().setAppName("App_Name")     .setMaster("spark://localhost:18080").set("spark.ui.port","18080"); 

回答2:

The standard port for master is 7077 not 18080. Maybe you can try 7077.

文章来源: Could not parse Master URL: 'spark:http://localhost:18080'