Task not serializable: java.io.NotSerializableException when calling function outside closure only on classes not objects

后端 未结 9 1708
悲&欢浪女
悲&欢浪女 2020-11-22 05:29

Getting strange behavior when calling function outside of a closure:

  • when function is in a object everything is working
  • when function is in a class ge
9条回答
  •  萌比男神i
    2020-11-22 05:49

    Complete talk fully explaining the problem, which proposes a great paradigm shifting way to avoid these serialization problems: https://github.com/samthebest/dump/blob/master/sams-scala-tutorial/serialization-exceptions-and-memory-leaks-no-ws.md

    The top voted answer is basically suggesting throwing away an entire language feature - that is no longer using methods and only using functions. Indeed in functional programming methods in classes should be avoided, but turning them into functions isn't solving the design issue here (see above link).

    As a quick fix in this particular situation you could just use the @transient annotation to tell it not to try to serialise the offending value (here, Spark.ctx is a custom class not Spark's one following OP's naming):

    @transient
    val rddList = Spark.ctx.parallelize(list)
    

    You can also restructure code so that rddList lives somewhere else, but that is also nasty.

    The Future is Probably Spores

    In future Scala will include these things called "spores" that should allow us to fine grain control what does and does not exactly get pulled in by a closure. Furthermore this should turn all mistakes of accidentally pulling in non-serializable types (or any unwanted values) into compile errors rather than now which is horrible runtime exceptions / memory leaks.

    http://docs.scala-lang.org/sips/pending/spores.html

    A tip on Kryo serialization

    When using kyro, make it so that registration is necessary, this will mean you get errors instead of memory leaks:

    "Finally, I know that kryo has kryo.setRegistrationOptional(true) but I am having a very difficult time trying to figure out how to use it. When this option is turned on, kryo still seems to throw exceptions if I haven't registered classes."

    Strategy for registering classes with kryo

    Of course this only gives you type-level control not value-level control.

    ... more ideas to come.

提交回复
热议问题