Spark Kryo register for array class

问题

I am trying to register a class with array (Spark Java with Kryo activated), log shows a clear message:

Class is not registered: org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[]

I have written several combinations, but these do not work:

        kryo.register(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[]")); // ERROR
        kryo.register(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[].class")); // ERROR
        kryo.register(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation$Array")); // ERROR
        kryo.register(Class.forName("[org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation")); // ERROR
        kryo.register(Class.forName("[Lorg.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation")); // ERROR
        kryo.register(Class.forName("Array[org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation]")); // ERROR
        kryo.register(Class.forName("[[org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation"));  // ERROR

I also tried to write a registration class without Class.forName but Java cannot resolve the symbol InMemoryFileIndex$SerializableBlockLocation:

kryo.register(org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[].class);

All other classes work in my KryoRegister.class.

回答1:

I found a similar question here, try:

kryo.register(Array.newInstance(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation"), 0)
                        .getClass());

来源：https://stackoverflow.com/questions/54106392/spark-kryo-register-for-array-class

标签

java

apache-spark

kryo