Task not serializable when using object in REPL

China☆狼群 提交于 2019-12-01 06:32:08

问题


So, another SO question prompted me to try the following:

object Foo{
  def f = 1
}

sc.parallelize(List(1)).map(x=>{
  val myF = Foo.f _
  x + myF()
}

Which works, but the following does not:

object Foo{
  def f = 1
  def run(rdd: org.apache.spark.rdd.RDD[Int]) = rdd.map(x=>{
    val myF = Foo.f _
    x + myF()
  }
}

Foo.run(sc.parallelize(List(1)))

I will take a deeper look at the serialization stack tomorrow when I can remove the mess of the REPL output. But, this should be the same thing. Why does one way yell and the other does not.

来源:https://stackoverflow.com/questions/29762964/task-not-serializable-when-using-object-in-repl

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!