MapWithStateRDDRecord with kryo

[亡魂溺海] 提交于 2021-02-08 07:54:05

问题


How can I register MapWithStateRDDRecord in kryo?

When I'm trying to do.

`sparkConfiguration.registerKryoClasses(Array(classOf[org.apache.spark.streaming.rdd.MapWithStateRDD))`

I get an error

class MapWithStateRDDRecord in package rdd cannot be accessed in package org.apache.spark.streaming.rdd
[error]       classOf[org.apache.spark.streaming.rdd.MapWithStateRDDRecord]

I'd like to make sure that all serialization is done via kryo thus I set SparkConf().set("spark.kryo.registrationRequired", "true"). With this setting enabled I get exceptions during runtime. java.lang.IllegalArgumentException (Class is not registered: org.apache.spark.streaming.rdd.MapWithStateRDDRecord)

来源:https://stackoverflow.com/questions/42555183/mapwithstaterddrecord-with-kryo

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!