MLlib MatrixFactorizationModel recommendProducts(user, num) failing on some users

问题

I trained a MatrixFactorizationModel model using ALS.train() and now using model.recommendProducts(user, num) to get the top recommended products, but the code fails on some users with the following error:

  user_products = model.call("recommendProducts", user, prodNum)
  File "/usr/lib/spark/python/pyspark/mllib/common.py", line 136, in call
    return callJavaFunc(self._sc, getattr(self._java_model, name), *a)
  File "/usr/lib/spark/python/pyspark/mllib/common.py", line 113, in callJavaFunc
    return _java2py(sc, func(*args))
  File "/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in __call__
  File "/usr/lib/spark/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o68.recommendProducts.
: java.util.NoSuchElementException: next on empty iterator
    at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
    at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
    at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:64)
    at scala.collection.IterableLike$class.head(IterableLike.scala:91)
    at scala.collection.mutable.WrappedArray.scala$collection$IndexedSeqOptimized$$super$head(WrappedArray.scala:34)
    at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOptimized.scala:120)
    at scala.collection.mutable.WrappedArray.head(WrappedArray.scala:34)
    at org.apache.spark.mllib.recommendation.MatrixFactorizationModel.recommendProducts(MatrixFactorizationModel.scala:117)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
    at py4j.Gateway.invoke(Gateway.java:259)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:207)
    at java.lang.Thread.run(Thread.java:745)

As you can see in first line above, I am running

user_products = model.call("recommendProducts", user, prodNum)

instead of

user_products = model.recommendProducts(user, prodNum)

because the latter is not implemented in 1.3.0 pyspark which I am using. Anyhow, it correctly returns prediction for some users, but then it fails on others.

I understand that it probably does not have the exact number of predictions I am requesting, I would expect it would return fewer.

回答1:

Short answer:

You have trained model on ratings where userID was in range [0;N]
You have asked for a recommendation for a userID=N+x, where x is a positive integer number. This caused the exception.

The rest of the answer:

You are allowed to ask for recommendations within the userID space used during the training phase
You are allowed to ask for as many recommendations as the product space used during the training phase

When you want to make recommendation for a new user you have to add information about his or her taste to a training dataset.

You can find examples covering Collaborative Filtering in Spark Summit 2014 Hands-on Exercises.

(If I missed anything please correct me in comment below)

来源：https://stackoverflow.com/questions/32488328/mllib-matrixfactorizationmodel-recommendproductsuser-num-failing-on-some-user

标签

apache-spark

apache-spark-mllib

collaborative-filtering

matrix-factorization