Spark ALS transform performance

Question

I'm a spark newbie. I'm using pyspark for ALS recommendation. The fitting takes few minutes and runs fairly quickly. However the model.transform function takes a long time and requires significantly more nodes in the cluster.

I was wondering if there are any optimization I can do to deal with the model.transform function?
What is the method used underneath? Is it just simple matrix multiplication? If so can't I use another matrix multiplication library just for that?

addmeaning addmeaning · Accepted Answer · 2018-02-28T08:48:44

Yes, the essence of ALS algorithm is matrix multiplication. First thing to check if you have this kind of warning in your log

15/06/15 11:17:36 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
15/06/15 11:17:36 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
15/06/15 11:17:36 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
15/06/15 11:17:36 WARN LAPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK

If you do — you should probably add external linear algebra library as a dependency. Refer to this guide

Spark ALS transform performance

1 Answers