Scala member field visibility in Spark jobs

你说的曾经没有我的故事 提交于 2019-12-19 10:15:42

问题


I have a Scala class that I define like so:

import org.apache.spark.{SparkConf, SparkContext}

object TestObject extends App{
  val FAMILY = "data".toUpperCase

  override def main(args: Array[String]) {
    val sc = new SparkContext(new SparkConf())

    sc.parallelize(1 to 10)
      .map(getData)
      .saveAsTextFile("my_output")
  }

  def getData(i: Int) = {
    ( i, FAMILY, "data".toUpperCase )
  }
}

I submit it to a YARN cluster like so:

HADOOP_CONF_DIR=/etc/hadoop/conf spark-submit \
    --conf spark.hadoop.validateOutputSpecs=false \
    --conf spark.yarn.jar=hdfs:/apps/local/spark-assembly-1.2.1-hadoop2.4.0.jar \
    --deploy-mode=cluster \
    --master=yarn \
    --class=TestObject \
    target/scala-2.11/myjar-assembly-1.1.jar

Unexpectedly, the output looks like the following, indicating that the getData method can't see the value of FAMILY:

(1,null,DATA)
(2,null,DATA)
(3,null,DATA)
(4,null,DATA)
(5,null,DATA)
(6,null,DATA)
(7,null,DATA)
(8,null,DATA)
(9,null,DATA)
(10,null,DATA)

What do I need to understand, about fields and scoping and visibility and spark submission and objects and singletons and whatnot, to understand why this is happening? And what should I be doing instead, if I basically want variables defined as "constants" visible to the getData method?


回答1:


I might be missing something, but I don't think you should be defining a main method. When you extend App, you inherit a main, and you should not override it since that is what actually invokes the code in your App.

For example, the simple class in your answer should be written

object TestObject extends App {
  val FAMILY = "data"
  println(FAMILY, "data")
}



回答2:


Figured it out. It's the App trait causing trouble. It manifests even in this simple class:

object TestObject extends App {
  val FAMILY = "data"
  override def main(args: Array[String]) = println(FAMILY, "data")
}
# prints "(null,data)"

Apparently App inherits from DelayedInit, which means that when main() runs, FAMILY hasn't been initialized. Exactly what I don't want, so I'm going to stop using App.



来源:https://stackoverflow.com/questions/29133161/scala-member-field-visibility-in-spark-jobs

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!