Spark CountVectorizer return udt instead of vector [duplicate]

问题

I try to create a vector of token counts for a LDA analysis in Spark 2.3.0. I have followed some tutorial and at each time they use CountVectorizer to easily convert Array of String to Vector.

I run this short example on my Databricks notebook :

import org.apache.spark.ml.feature.CountVectorizer

val testW = Seq(
  (8, Array("Zara", "Nuha", "Ayan", "markle")),
  (9, Array("fdas", "test", "Ayan", "markle")),
  (10, Array("qwertzu", "test", "Ayan", "fdaf"))
  ).toDF("id", "filtered")

// Set params for CountVectorizer
val vectorizer = new CountVectorizer()
  .setInputCol("filtered")
  .setOutputCol("features")
  .setVocabSize(5) 
  .setMinDF(2) 
  .fit(testW)

// Create vector of token counts
val articlesCountVector = vectorizer.transform(testW).select("id", "features")
display(articlesCountVector)

and the output is the following : output

But in all tutorial I have read, the type of "features" is vector. Why in my case is it udt ?

Did i forget something ? Why it is not a vector ?

Is it possible to convert it ? because I cannot create LDA model with this udt type.

回答1:

There is no issue here. What is you see, is the detail of implementation of the Databricks display functions.

Internally, both o.a.s.ml.linalg.Vector and o.a.s.mllib.linalg.Vector are not natively represented in the Dataset API, and use UDTs (UserDefinedTypes). Hence the output.

You can find the exact meaning of all fields in Understanding Output of VectorAssembler --- Spark

来源：https://stackoverflow.com/questions/50567836/spark-countvectorizer-return-udt-instead-of-vector

标签

apache-spark

apache-spark-sql

apache-spark-mllib