java.lang.UnsupportedOperationException: Schema for type MyClass is not supported

Question

I am using Spark 1.5.0 and I have an issue while creating a dataframe from my rdd.

Here is the code:

case class MyC (myclass: MyClass) 

  val df = r         


        

Answer 1

Spark uses reflection to infer dataframe schema, but cannot do so for arbitrary classes. I'm not sure if I can state an answer better than the Spark docs can, and there's a section dedicated to exactly this.

To be specific, the problem is that there are a limited number of types for which Spark can infer schema. Ints, Strings, Seqs/Arrays are all supported (as well as case classes containing elements of these types), but an arbitrary class MyClass is not.