apache-spark-mllib

How can you evaluate the implicit feedback collaborative filtering algorithm of Apache Spark, given that the implicit "ratings" can vary from zero to anything, so a simple MSE or RMSE does not have much meaning? To answer this question, you'll need to go back to the original paper that defined what is implicit feedback and the ALS algorithm Collaborative Filtering for Implicit Feedback Datasets by Yifan Hu, Yehuda Koren and Chris Volinsky . What is implicit feedback ? In the absence of explicit ratings, recommender systems can infer user preferences from the more abundant implicit feedback ,

Do you guys know where can I find examples of multiclass classification in Spark. I spent a lot of time searching in books and in the web, and so far I just know that it is possible since the latest version according the documentation. zero323 ML ( Recommended in Spark 2.0+ ) We'll use the same data as in the MLlib below. There are two basic options. If Estimator supports multilclass classification out-of-the-box (for example random forest) you can use it directly: val trainRawDf = trainRaw.toDF import org.apache.spark.ml.feature.{Tokenizer, CountVectorizer, StringIndexer} import org.apache

What .map() function in python do I use to create a set of labeledPoints from a spark dataframe? What is the notation if The label/outcome is not the first column but I can refer to its column name, 'status'? I create the Python dataframe with this .map() function: def parsePoint(line): listmp = list(line.split('\t')) dataframe = pd.DataFrame(pd.get_dummies(listmp[1:]).sum()).transpose() dataframe.insert(0, 'status', dataframe['accepted']) if 'NULL' in dataframe.columns: dataframe = dataframe.drop('NULL', axis=1) if '' in dataframe.columns: dataframe = dataframe.drop('', axis=1) if 'rejected'

I build a simple recommendation system for the MovieLens DB inspired by https://databricks-training.s3.amazonaws.com/movie-recommendation-with-mllib.html . I also have problems with explicit training like here: Apache Spark ALS collaborative filtering results. They don't make sense Using implicit training (on both explicit and implicit data) gives me reasonable results, but explicit training doesn't. While this is ok for me by now, im curious on how to update a model. While my current solution works like having all user ratings generate model get recommendations for user I want to have a flow

I have a column in my Spark DataFrame: |-- topics_A: array (nullable = true) | |-- element: string (containsNull = true) I'm using CountVectorizer on it: topic_vectorizer_A = CountVectorizer(inputCol="topics_A", outputCol="topics_vec_A") I get NullPointerExceptions, because sometimes the topic_A column contains null. Is there a way around this? Filling it with a zero-length array would work ok (although it will blow out the data size quite a lot) - but I can't work out how to do a fillNa on an Array column in PySpark. Personally I would drop columns with NULL values because there is no useful

Im new in spark and Machine learning in general. I have followed with success some of the Mllib tutorials, i can't get this one working: i found the sample code here : https://spark.apache.org/docs/latest/mllib-linear-methods.html#linear-least-squares-lasso-and-ridge-regression (section LinearRegressionWithSGD) here is the code: import org.apache.spark.mllib.regression.LabeledPoint import org.apache.spark.mllib.regression.LinearRegressionModel import org.apache.spark.mllib.regression.LinearRegressionWithSGD import org.apache.spark.mllib.linalg.Vectors // Load and parse the data val data = sc

I noticed that the ml StandardScaler does not attach metadata to the output column: import org.apache.spark.ml.Pipeline import org.apache.spark.ml.feature._ val df = spark.read.option("header", true) .option("inferSchema", true) .csv("/path/to/cars.data") val strId1 = new StringIndexer() .setInputCol("v7") .setOutputCol("v7_IDX") val strId2 = new StringIndexer() .setInputCol("v8") .setOutputCol("v8_IDX") val assmbleFeatures: VectorAssembler = new VectorAssembler() .setInputCols(Array("v0", "v1", "v2", "v3", "v4", "v5", "v6", "v7_IDX")) .setOutputCol("featuresRaw") val scalerModel = new