Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects

Question

Getting strange behavior when calling function outside of a closure:

• when function is in a object everything is working
• when function is in a class ge

Answer 1

I had a similar experience.

The error was triggered when I initialize a variable on the driver (master), but then tried to use it on one of the workers. When that happens, Spark Streaming will try to serialize the object to send it over to the worker, and fail if the object is not serializable.

I solved the error by making the variable static.

Previous non-working code

  private final PhoneNumberUtil phoneUtil = PhoneNumberUtil.getInstance();

Working code

  private static final PhoneNumberUtil phoneUtil = PhoneNumberUtil.getInstance();

Credits:

1. https://docs.microsoft.com/en-us/answers/questions/35812/sparkexception-job-aborted-due-to-stage-failure-ta.html ( The answer of pradeepcheekatla-msft)
2. https://databricks.gitbooks.io/databricks-spark-knowledge-base/content/troubleshooting/javaionotserializableexception.html