apache-spark-mllib

I am preparing a DataFrame with an id and a vector of my features to be used later for doing predictions. I do a groupBy on my dataframe, and in my groupBy I am merging couple of columns as lists into a new column: def mergeFunction(...) // with 14 input variables val myudffunction( mergeFunction ) // Spark doesn't support this df.groupBy("id").agg( collect_list(df(...)) as ... ... // too many of these (something like 14 of them) ).withColumn("features_labels", myudffunction( col(...) , col(...) ) .select("id", "feature_labels") This is how I am creating my feature vectors and their labels. It

I'm trying to figure out how an ALS model can predict values for new users in between it being updated by a batch process. In my search, I came across this stackoverflow answer . I've copied the answer below for the reader's convenience: You can get predictions for new users using the trained model (without updating it): To get predictions for a user in the model, you use its latent representation (vector u of size f (number of factors)), which is multiplied by the product latent factor matrix (matrix made of the latent representations of all products, a bunch of vectors of size f) and gives

I wanted to convert the spark data frame to add using the code below: from pyspark.mllib.clustering import KMeans spark_df = sqlContext.createDataFrame(pandas_df) rdd = spark_df.map(lambda data: Vectors.dense([float(c) for c in data])) model = KMeans.train(rdd, 2, maxIterations=10, runs=30, initializationMode="random") The detailed error message is: --------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-11-a19a1763d3ac> in <module>() 1 from pyspark.mllib.clustering import KMeans 2 spark_df = sqlContext

KMeans has several parameters for its training , with initialization mode defaulted to kmeans||. The problem is that it marches quickly (less than 10min) to the first 13 stages, but then hangs completely , without yielding an error! Minimal Example which reproduces the issue (it will succeed if I use 1000 points or random initialization): from pyspark.context import SparkContext from pyspark.mllib.clustering import KMeans from pyspark.mllib.random import RandomRDDs if __name__ == "__main__": sc = SparkContext(appName='kmeansMinimalExample') # same with 10000 points data = RandomRDDs