VectorUDT usage

Question

I have to get the datatype and do a case match and convert it to some required format. But the usage of org.apache.spark.ml.linalg.VectorUDT is showing Ve

Answer 1

For org.apache.spark.ml.linalg types you should specify schema using org.apache.spark.ml.linalg.SQLDataTypes which provide singleton instances of the private UDT types:

• MatrixType for matrices (org.apache.spark.ml.linalg.Matrix).

  scala> org.apache.spark.ml.linalg.SQLDataTypes.MatrixType.getClass
  res0: Class[_ <: org.apache.spark.sql.types.DataType] = class org.apache.spark.ml.linalg.MatrixUDT
  

• VectorType for vectors (org.apache.spark.ml.linalg.Vector).

  scala> org.apache.spark.ml.linalg.SQLDataTypes.VectorType.getClass
  res1: Class[_ <: org.apache.spark.sql.types.DataType] = class org.apache.spark.ml.linalg.VectorUDT