Remote Spark Job fails: No assemblies found

问题

I am running a Spark job in a Vanilla Spark (Datastax) with this conf:

val conf: SparkConf = new SparkConf()
  .setAppName("Fuzzy")
  .setMaster("spark://127.0.0.1:7077")
  .set("spark.cores.max", "2")
  .setJars(Seq("my-jar"))

val sc: SparkContext = SparkContext.getOrCreate(conf)


val NUM_SAMPLES: Int = 500

val join = sc.parallelize(1 to NUM_SAMPLES).filter { _ =>
  val x = math.random
  val y = math.random
  x*x + y*y < 1
}.count()
println(s"Pi is roughly ${4.0 * join / NUM_SAMPLES}")

This is an Spark example.

When I run with with local[2] it's fine, but when I use a external Spark it fails.

17/04/24 14:44:20 INFO SparkDeploySchedulerBackend: Asked to remove non-existent executor 139 17/04/24 14:44:20 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170424144418-0012/140 on hostPort 127.0.0.1:51227 with 2 cores, 512.0 MB RAM 17/04/24 14:44:20 INFO AppClient$ClientEndpoint: Executor updated: app-20170424144418-0012/140 is now RUNNING 17/04/24 14:44:20 INFO AppClient$ClientEndpoint: Executor updated: app-20170424144418-0012/140 is now FAILED (java.lang.IllegalStateException: No assemblies found in '/Users/jta/Downloads/dse-5.0.6-2/dse-5.0.6/resources/spark/lib'.) 17/04/24 14:44:20 INFO SparkDeploySchedulerBackend: Executor app-20170424144418-0012/140 removed: java.lang.IllegalStateException: No assemblies found in '/Users/jta/Downloads/dse-5.0.6-2/dse-5.0.6/resources/spark/lib'. 17/04/24 14:44:20 INFO BlockManagerMaster: Removal of executor 140 requested 1

回答1:

I already know what the problem was. I ran different versions of Spark/Datastax and all of them used the same folders. Somehow, Spark got confused about it and stop doing that and showing that error.

Once I've run it from Docker, everything went fine.

来源：https://stackoverflow.com/questions/43590062/remote-spark-job-fails-no-assemblies-found