python - Multiprocessing scikit-learn

Question

I got linearsvc working against training set and test set using load_file method i am trying to get It working on Multiprocessor enviorment.

How can i get multiprocessing work on LinearSVC().fit() LinearSVC().predict()? I am not really familiar with datatypes of scikit-learn yet.

I am also thinking about splitting samples into multiple arrays but i am not familiar with numpy arrays and scikit-learn data structures.

Doing this it will be easier to put into multiprocessing.pool() , with that , split samples into chunks , train them and combine trained set back later , would it work ?

EDIT: Here is my scenario:

lets say , we have 1 million files in training sample set , when we want to distribute processing of Tfidfvectorizer on several processors we have to split those samples (for my case it will only have two categories , so lets say 500000 each samples to train) . My server have 24 cores with 48 GB , so i want to split each topics into number of chunks 1000000 / 24 and process Tfidfvectorizer on them. Like that i would do to Testing sample set , as well as SVC.fit() and decide(). Does it make sense?

Thanks.

PS: Please do not close this .

Correct me if I'm wrong, but an SVM usually doesn't take long to make a decision. It might make more sense to perform the decoding for different samples in parallel than to parallelize the decoding for one sample. — Qnan
what if i am going to do that on 21 million documents? Would it take long? — Phyo Arkar Lwin
I am thinking about different samples too , is it able to re-combine different samples after spliting them for each process? — Phyo Arkar Lwin
I don't think I get your question. The samples are independent. Why do you have to re-combine something? — Qnan
Tfidfvectorizer is not parallelizable be case of the central vocabulary. We either need a shared vocabulary (e.g. using a redis server on the cluster) or implement a HashVectorizer that does not exists yet. — ogrisel

Unknown Unknown · Accepted Answer · 2012-10-26T07:36:25

I think using SGDClassifier instead of LinearSVC for this kind of data would be a good idea, as it is much faster. For the vectorization, I suggest you look into the hash transformer PR.

For the multiprocessing: You can distribute the data sets across cores, do partial_fit, get the weight vectors, average them, distribute them to the estimators, do partial fit again.

Doing parallel gradient descent is an area of active research, so there is no ready-made solution there.

How many classes does your data have btw? For each class, a separate will be trained (automatically). If you have nearly as many classes as cores, it might be better and much easier to just do one class per core, by specifying n_jobs in SGDClassifier.

python - Multiprocessing scikit-learn

2 Answers