Spark mllib svd gives: Java OutOfMemory Error

Question

I am using the svd library of mllib to do some dimensionality reduction on a big matrix:

the data is about 20G,
and the spark memory is 60G

I got the following warning and error message:

WARN ARPACK: Failed to load implementation from: com.github.fommil.netlib.NativeSystemARPACK WARN ARPACK: Failed to load implementation from: com.github.fommil.netlib.NativeRefARPACK WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:2766) 
at org.apache.spark.mllib.linalg.EigenValueDecomposition$.symmetricEigs(EigenValueDecomposition.scala:128)

at org.apache.spark.mllib.linalg.distributed.RowMatrix.computeSVD(RowMatrix.scala:258)

at org.apache.spark.mllib.linalg.distributed.RowMatrix.computeSVD(RowMatrix.scala:190)

To resolve the warning messages, I build Spark (1.2) with the command -Pnetlib-lgpl locally, and the warning messages disappeared when I test locally. And the log shows the netlib library is working properly:

15/03/05 20:07:03 INFO JniLoader: successfully loaded /tmp/jniloader7217840327264308862netlib-native_system-linux-x86_64.so 15/03/05 20:07:11 INFO JniLoader: already loaded netlib-native_system-linux-x86_64.so

Then I installed the Spark (1.2) compiled with -Pnetlib-lgpl on AWS EMR, but the warning and error message still shows up.

I was not sure whether it is the problem of my local compilation, so I compiled Spark (1.2) on AWS EC2 and installed it on AWS EMR, but the warning and error message still shows up.

Could anyone tell me how to solve this problem? Much appreciated!

Did you perform the profile compilation on all nodes in the cluster, or just the master node? — Dan Osipov
What do you mean by "Spark memory is 60G" -- you are running a non distributed operation so only driver memory matters — Sean Owen
@DanOsipov I only performed the compilation on the master node. Should I perform the profile compilation on all nodes in the cluster, and how can I do that? — Jiang Xiang
@SeanOwen I am running distributed operation on 10 machine with 6G memory on each. I did not specify the driver and worker memories. Should I set this? — Jiang Xiang
The SVD operation you are calling happens on the driver. The covariance matrix is calculated on the cluster though. Where are you running out of memory? Driver right? — Sean Owen

Jiang Xiang Jiang Xiang · Accepted Answer · 2015-03-11T17:33:24

The problem is partially solved. Thanks for the comments by @SeanOwen.

The reason why I am getting the java memory error is because, the computation for top eigenvectors are on the driver, so I need to make sure that I have enough memory on the driver node.

When using spark-submit with --driver-memory 5G, the problem is solved.

But the warning messages still exist.

Spark mllib svd gives: Java OutOfMemory Error

1 Answers