Recommendation based on implicit feedback - Spark Mlib

Question

I have implicit feedback from users about their interaction with different products. Following is the structure of dataset: user_id, product_category, event_date,view_count,purchase_count

Based on this data I am trying to predict a score for a user's affinity to a product category. MLIB implicit form expects the input dataset of the format Rating(user: Int, product: Int, rating: Double). But how do I include view_count and purchase_count ?

If MLLIB can't be used is there any other method I can use. My product categories are limited (only 8 product categories). In that case can I use 8 logistic regression models to predict affinity ?

Thanks

Answer 1

What you're describing is called multinomial logistic regression which is used for multiclass classification. This is a generalization of the typical logistic regression which has only 2 outcomes (binary).

It looks like you were trying to apply a collaborative filtering model which won't work in this case as you have pointed out.

Check out the Spark ML/MLlib docs for multinomial logistic regression.

You would treat each of the inputs (view_count and purchase_count) as separate features (input data) to train the model using existing, known data including the outcomes.

When you score/predict on new data, the model will return a probability for each of the possible outcomes: 8, in this case ... 1 for each of the product categories. Then choose the category with the highest probability.

Below are some relevant Spark Documentation links:

Spark's Logistic Regression Programming Guide

http://spark.apache.org/docs/latest/mllib-linear-methods.html#logistic-regression

Current spark.mllib Logistic Regression API

http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.mllib.classification.LogisticRegressionModel

http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.mllib.evaluation.MulticlassMetrics

New spark.ml Logistic Regression API

(Note: this only supports binary, logistic regression right now, but you should migrate to this when it's available)

http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.classification.LogisticRegression

http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator

Answer 2

Are you trying to predict affinity score or go ahead with recommendation based on affinity? For example, you might have a ratio of purchase to visit, keep that as rating (affinity rating here) and perform recommendation