问题
I am running into an exception while inside KMeans.train() like below:
java.lang.IllegalArgumentException: requirement failed
at scala.Predef$.require(Predef.scala:212)
at org.apache.spark.mllib.util.MLUtils$.fastSquaredDistance(MLUtils.scala:487)
at org.apache.spark.mllib.clustering.KMeans$.fastSquaredDistance(KMeans.scala:589)
at org.apache.spark.mllib.clustering.KMeans$$anonfun$runAlgorithm$3.apply(KMeans.scala:304)
at org.apache.spark.mllib.clustering.KMeans$$anonfun$runAlgorithm$3.apply(KMeans.scala:301)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:99)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:230)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:40)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:99)
at org.apache.spark.mllib.clustering.KMeans.runAlgorithm(KMeans.scala:301)
at org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:227)
at org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:209)
at org.apache.spark.mllib.clustering.KMeans$.train(KMeans.scala:530)
This doesn't give me any clue on where to start debugging.
I found an old post but that issue was in KMeans.predict() whereas this is happening in the training phase itself.
回答1:
Just take a look at the source code and it will become clear:
- Your vectors have to have the same size.
- The norms of both vectors should be non-negative.
https://github.com/apache/spark/blob/17af727e38c3faaeab5b91a8cdab5f2181cf3fc4/mllib/src/main/scala/org/apache/spark/mllib/util/MLUtils.scala#L500
private[mllib] def fastSquaredDistance(
v1: Vector,
norm1: Double,
v2: Vector,
norm2: Double,
precision: Double = 1e-6): Double = {
val n = v1.size
require(v2.size == n)
require(norm1 >= 0.0 && norm2 >= 0.0)
...
回答2:
The most possible cause of this error is that the input values in the vector are of varying dimensions. We can be more certain on this reason if you can share the details of the inputs (vector inputs) being passed.
Please recheck that vectors being passed are of same dimensions.
回答3:
Some of your rows has null values, use: "dropna" or equivalent
来源：https://stackoverflow.com/questions/46968523/spark-illegalargumentexception-in-kmeans-train