Is scikit-learn suitable for big data tasks?

Question

I'm working on a TREC task involving use of machine learning techniques, where the dataset consists of more than 5 terabytes of web documents, from which bag-of-words vectors are planned to be extracted. scikit-learn has a nice set of functionalities that seems to fit my need, but I don't know whether it is going to scale well to handle big data. For example, is HashingVectorizer able to handle 5 terabytes of documents, and is it feasible to parallelize it? Moreover, what are some alternatives out there for large-scale machine learning tasks?

Yeah, scikit-learn will give you the best answers...logistic regression should work with what you need as well — Ryan Saxe
LogisticRegression as implemented in scikit-learn won't work on such a big dataset: this is a wrapper for liblinear that requires to load the data in memory prior to fitting. — ogrisel
@ogrisel, LogisticRegression in sklearn uses 2nd order optimization methods, so not well suited to large scale data. And SGDClassifier could be eligible, but how do you do feature scaling on such large scale dataset? — avocado

ogrisel ogrisel · Accepted Answer · 2013-06-10T07:24:44

HashingVectorizer will work if you iteratively chunk your data into batches of 10k or 100k documents that fit in memory for instance.

You can then pass the batch of transformed documents to a linear classifier that supports the partial_fit method (e.g. SGDClassifier or PassiveAggressiveClassifier) and then iterate on new batches.

You can start scoring the model on a held-out validation set (e.g. 10k documents) as you go to monitor the accuracy of the partially trained model without waiting for having seen all the samples.

You can also do this in parallel on several machines on partitions of the data and then average the resulting coef_ and intercept_ attribute to get a final linear model for the all dataset.

I discuss this in this talk I gave in March 2013 at PyData: http://vimeo.com/63269736

There is also sample code in this tutorial on paralyzing scikit-learn with IPython.parallel taken from: https://github.com/ogrisel/parallel_ml_tutorial

Is scikit-learn suitable for big data tasks?

1 Answers