问题
So, another SO question prompted me to try the following:
object Foo{
def f = 1
}
sc.parallelize(List(1)).map(x=>{
val myF = Foo.f _
x + myF()
}
Which works, but the following does not:
object Foo{
def f = 1
def run(rdd: org.apache.spark.rdd.RDD[Int]) = rdd.map(x=>{
val myF = Foo.f _
x + myF()
}
}
Foo.run(sc.parallelize(List(1)))
I will take a deeper look at the serialization stack tomorrow when I can remove the mess of the REPL output. But, this should be the same thing. Why does one way yell and the other does not.
来源:https://stackoverflow.com/questions/29762964/task-not-serializable-when-using-object-in-repl