Scikit learn Naive Bayes ValueError: dimension mismatch

Question

I am working on Naive Bayes classifier in Scikit-learn.

both during the training and predict phase I use following code to get csr_matrix from list of tuples:

def convert_to_csr_matrix(vectors):
    """
    convert list of tuples representation to scipy csr_matrix that is needed
    for scikit learner
    """
    logger.info("building the csr_sparse matrix representing tf-idf")
    row = [[i] * len(v) for i, v in enumerate(vectors)]
    row = list(chain(*row))
    column = [j for j, _ in chain(*vectors)]
    data = [d for _, d in chain(*vectors)]
    return csr_matrix((data, (row, column)))

Which I have implemented mostly based on scipy csr_matrix from several vectors represented as list of sets

Unfortunately now during the predict phase I am getting the following error:

File "/Users/zikes/project/taxonomy_data_preprocessing/single_classification.py", line 93, in predict
top_predictions = self.top.predict(item)
File "/Users/zikes/project/taxonomy_data_preprocessing/single_classification.py", line 124, in predict
category, res = model.predict(item)
File "/Users/zikes/project/taxonomy_data_preprocessing/single_classification.py", line 176, in predict
prediction = self.clf.predict(item)
File "/Users/zikes/.virtualenvs/taxonomy/lib/python2.7/site-packages/sklearn/naive_bayes.py", line 64, in predict
jll = self._joint_log_likelihood(X)
File "/Users/zikes/.virtualenvs/taxonomy/lib/python2.7/site-packages/sklearn/naive_bayes.py", line 615, in _joint_log_likelihood
return (safe_sparse_dot(X, self.feature_log_prob_.T)
File "/Users/zikes/.virtualenvs/taxonomy/lib/python2.7/site-packages/sklearn/utils/extmath.py", line 178, in safe_sparse_dot
ret = a * b
File "/Users/zikes/.virtualenvs/taxonomy/lib/python2.7/site-packages/scipy/sparse/base.py", line 354, in __mul__
raise ValueError('dimension mismatch')
ValueError: dimension mismatch

Does anyone has idea what can be wrong? I guess that somehow sparse vectors have wrong dimensions. But I don't see why?

During the debugging I have printed out in the log mentioned feature_log_prob_ from Naive Bayes model and it looks as:

[[-11.82052115 -12.51735721 -12.51735721 ..., -12.51735721 -11.60489688
-12.2132116 ]
[-12.21403023 -12.51130295 -12.51130295 ..., -11.84156341 -12.51130295
-12.51130295]]

And shape: (2, 53961)

My to predict csr_matrix = (0, 7637) 0.770238101052 (0, 21849) 0.637756432886

And represented as list of tuples it looks as: [(7637, 0.7702381010520318), (21849, 0.6377564328862234)]

ziky90 ziky90 · Accepted Answer · 2015-08-05T15:12:56

So after a bit of investigation of the problem I have realised that possible fix might be in method:

def convert_to_csr_matrix(vectors):
   """
   convert list of tuples representation to scipy csr_matrix that is needed
   for scikit learner
   """
   logger.info("building the csr_sparse matrix representing tf-idf")
   row = [[i] * len(v) for i, v in enumerate(vectors)]
   row = list(chain(*row))
   column = [j for j, _ in chain(*vectors)]
   data = [d for _, d in chain(*vectors)]
   return csr_matrix((data, (row, column)))

Line return csr_matrix((data, (row, column))) should be replaced by return csr_matrix((data, (row, column)), shape=(len(vectors), dimension))

Scikit learn Naive Bayes ValueError: dimension mismatch

1 Answers