How to get sparse matrices into H2O?

7
votes

I am trying to get a sparse matrix into H2O and I was wondering whether that was possible. Suppose we have the following:

test <- Matrix(c(1,0,0,1,1,1,1,0,1), nrow = 3, sparse = TRUE)

and assuming my local H2O is localH2O, I can't seem to do the following:

as.h2o(test)

It gives the error: cannot coerce class "structure("dgCMatrix", package = "Matrix")" to a data.frame. That seems to be pretty logical, however assuming that test is so big that I can't transform it into a dataframe, how am I suppose to load this into H2O? Using a sparse matrix representation it is only 500MB or so.

How can I load a sparse matrix into H2O?

1
rsparse-matrixh2o

1 Answers

8
votes

It is cumbersome to transport data stored in R's memory to H2O's memory for essentially two reasons: R performs a POST of the file to stream up the data into H2O, which 1) doesn't take advantage of H2O's parallel reader, and 2) limits your data to existing in R.

Instead, make use of the h2o.importFile method from R to make use of H2O's parallel reader. Your data can live anywhere: HDFS, S3, regular filesystem...

H2O sports an SVMLight reader, so it is recommended to save your sparse Matrix from R in svmlight format.

Hope this helps!