How to create our custom feature extractor function and use it with countvectorizer in pipeline in Scikit-learn

Question

I am trying to create a simple question classifier using Scikit-learn. Currently I am able to classify question into corresponding classes using bag of word approach using countvectorizer function of Scikit. Now I want to create and add custom features with existing features generated with countvectorizer.

Suppose i want to create feature which checks if phone number is present in question or not and another feature which will extract length of question.

So what is the way to generate and merge all features together.

From this link i tried this template for custom feature extraction

`from sklearn.base import BaseEstimator, TransformerMixin

class SampleExtractor(BaseEstimator, TransformerMixin):

def __init__(self, vars):
    self.vars = vars  # e.g. pass in a column name to extract

def transform(self, X, y=None):
    return do_something_to(X, self.vars)  # where the actual feature extraction happens

def fit(self, X, y=None):
    return self  # generally does nothing`

but when when i put its output in pipeline with countvectorizer like

ppl = Pipeline([
('feats', FeatureUnion([
    ('ngram', CountVectorizer()), # can pass in either a pipeline
    ('ave', SampleExtractor()) # or a transformer
])),
('clf', LinearSVC())  # classifier

])

i get error

ValueError: blocks[0,:] has incompatible row dimensions

I think this error may be due to matrix of both feature are of not same dimension but I don't understand how to resolve it.

does your SampleExtractor() transform method reduce the number of rows? If so, that is not an acceptable transformation to put into a pipeline since only X is transformed, not y (and both X and y must have the same number of rows). — David

dkar dkar · Accepted Answer · 2016-07-06T20:17:01

Method transform of CountVectorizer returns a document-term matrix, the rows of which correspond to documents and the columns to terms. The (i,j)-th element of the matrix shows how many times the j-th term occurs in the i-th document. All you have to do is to add some more columns in this matrix, which correspond to the new features.

Example: Assume list doc_len contains the length of your documents in words and your document-term matrix is M. Then the code:

M_arr = M.toarray()
assert len(doc_len) == M_arr.shape[0]
np.append(M_arr,np.array(doc_len),axis=1)

will add a new column at the end of your matrix containing the new feature. You can repeat this process for additional new features. The extended matrix can be fed to whatever classifier you are using in the usual way.

How to create our custom feature extractor function and use it with countvectorizer in pipeline in Scikit-learn

1 Answers