Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects

Question

Getting strange behavior when calling function outside of a closure:

• when function is in a object everything is working
• when function is in a class ge

Answer 1

Grega's answer is great in explaining why the original code does not work and two ways to fix the issue. However, this solution is not very flexible; consider the case where your closure includes a method call on a non-Serializable class that you have no control over. You can neither add the Serializable tag to this class nor change the underlying implementation to change the method into a function.

Nilesh presents a great workaround for this, but the solution can be made both more concise and general:

def genMapper[A, B](f: A => B): A => B = {
  val locker = com.twitter.chill.MeatLocker(f)
  x => locker.get.apply(x)
}

This function-serializer can then be used to automatically wrap closures and method calls:

rdd map genMapper(someFunc)

This technique also has the benefit of not requiring the additional Shark dependencies in order to access KryoSerializationWrapper, since Twitter's Chill is already pulled in by core Spark