SciPy NumPy and SciKit-learn , create a sparse matrix

7
votes

I'm currently trying to classify text. My dataset is too big and as suggested here, I need to use a sparse matrix. My question is now, what is the right way to add an element to a sparse matrix? Let's say for example I have a matrix X which is my input .

X = np.random.randint(2, size=(6, 100))

Now this matrix X looks like an ndarray of an ndarray (or something like that).

If I do 

X2 = csr_matrix(X)

I have the sparse matrix, but how can I add another element to the sparce matrix ? for example this dense element: [1,0,0,0,1,1,1,0,...,0,1,0] to a sparse vector, how do I add it to the sparse input matrix ?

(btw, I'm very new at python, scipy,numpy,scikit ... everything)

1
pythonmatrixnumpyscipyscikit-learn
You should really read this: scikit-learn.org/dev/auto_examples/… – zenpoy
This is my second day working with python, that's a bit over the top for a second day to read. I found that too btw – Olivier_s_j
Some things simply take their time. Maybe you should invest some time into doing some tutorials on Python, Numpy, and Scipy. For example, in the answer in the other question I pointed you to some links, and zenpoy gave you another one. I assume you didn't read those links, since you posted this question mere minutes after I answered the other one. – HerrKaputt
I did read those, I even made a dummy example which works. Though the updating of the sparse matrix was not something I could find. If you don't know, I don't expect you to answer. – Olivier_s_j
@Ojtwist - You tagged your question under sklearn, so these are the answers you get. if you just asked how to concatenate two csr_matrix's you would get totally different answers... – zenpoy

1 Answers

14
votes

Scikit-learn has a great documentation, with great tutorials that you really should read before trying to invent it yourself. This one is the first one to read it explains how to classify text, step-by-step, and this one is a detailed example on text classification using sparse representation.

Pay extra attention to the parts where they talk about sparse representations, in this section. In general, if you want to use svm with linear kernel and you large amount of data, LinearSVC (which is based on Liblinear) is better.

Regarding your question - I'm sure there are many ways to concatenate two sparse matrices (btw this is what you should look for in google for other ways of doing it), here is one, but you'll have to convert from csr_matrix to coo_matrix which is anther type of sparse matrix: Is there an efficient way of concatenating scipy.sparse matrices?.

EDIT: When concatenating two matrices (or a matrix and an array which is a 1 dimenesional matrix) the general idea is to concatenate X1.data and X2.data and manipulate their indices and indptrs (or row and col in case of coo_matrix) to point to the correct places. Some sparse representations are better for specific operations and more complex for other operations, you should read about csr_matrix and see if this is the best representation. But I really urge you to start from those tutorials I posted above.