SVM Light loader for scikit learn

Question

I am running a bunch of experiments where the pre-processing changes on nearly every iteration. Once I have the data in a format, I need to run a classifier from scikits (liblinear). I can get the data into svm light format. I understand there is a module to convert that to the scikits standard. The issues is that I dont want to write a file to disk. I have 48 gigs of memory and would much rather make the conversion in memory.

Below is an example where I take the svm light format, write ito to a stringio object and sic the svm light loader on it. Unfortunately the loader refuses to be siced (it works other wise). Any suggestions? I am running over a 1000 different pre-processing pipelines, and i cant keep writing to the disk.

sample svm light data is at: http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/binary/a1a

My code

 import numpy as np
 from sklearn.datasets import svmlight_format

 import cStringIO

 bufFile = cStringIO.StringIO()
 with open('C:/Users/theDesktop/Desktop/sampleData.txt') as a:
        p = a.readlines()
        a.close()

 bufFile.writelines(p)

 X_train, y_train = svmlight_format.load_svmlight_file(bufFile)

My error:

Traceback (most recent call last):

File "C:\Users\theDesktop\Desktop\module1.py", 
line 25, in <module> X_train, y_train=svmlight_format.load_svmlight_file(bufFile)

File "C:\Python27\lib\site-packages\sklearn\datasets\svmlight_format.py", 
line 97, in load_svmlight_file zero_based))

File "C:\Python27\lib\site-packages\sklearn\datasets\svmlight_format.py", 
line 177, in load_svmlight_files for _, indices, _, _ in r):

File "C:\Python27\lib\site-packages\sklearn\datasets\svmlight_format.py", 
line 177, in <genexpr> for _, indices, _, _ in r):

File "C:\Python27\lib\site-packages\numpy\core\fromnumeric.py", 
line 1895, in amin return amin(axis, out)

ValueError: zero-size array to minimum.reduce without identity

clancularius clancularius · Accepted Answer · 2013-03-19T08:19:38

In a second look, I think I answered a different question below (If you are interested you can check) =) You are looking for a memory efficient format converter and a new perspective. First of all, what is problematic with having a 48 gigs of memory? It is plenty enough for text-processing unless you are working on a corpus including millions of documents. Second of all, if sklearn is writing files to disk as different files in different instances, then you can adjust it to overwrite and in the end you can remove the last iteration. Or, you can use os to connect terminal and from there by using rm command you can remove any file you like, in each iteration or all at once.

If the problem is python creating a new object in each iteration, then overwriting is again a solution for memory efficiency.

I am trying to help you; If I am misguiding or misunderstanding you please comment so that other people reading this do not get confused.

It is a bit late to the table, but as far as I understood you use sklearn as an interface between python and svm-light. I do not know what output you desire; but if it is just accuracy measures for varying pre-processing methods, then you can use the os library of python as an interface between the command line and then can call svm-light from there after installing it. Then by using grep or similar tools, you can extract accuracy or other desired info, since svm-light's learning and classification scripts do print log info to the the std output. Also note that svm-light outputs model file and other files after training and classification, thus you need to remove them with rm or some other tool in each iteration, or maybe just overwrite them again and again and remove them in the end.

By this way you can save the memory as you have said, and maybe(?) reach more info than you can from using sklearn. As a final remark, I am not sure about the complexity of this workaround: it may be more time-consuming, check it if time is a constraint for you.

I do not know if this is a possible and efficient workaround and I am sure you have already overcame the situation since it has been 3 months; but if you make a comment, this may help other people.

SVM Light loader for scikit learn

1 Answers