scikit classifier with unknown prediction

Question

I'm planning to use Python Scikit to do some text classification, and was planning to use using TfidfVectorizer and MultinomialNB.

but I realized that MultinomialNB will always predict my sample into an existing (known) category.

for example, if I have:

category A: trained with sample "this is green"
category B: trained with sample "this is blue"
category C: trained with sample "this is red"

and I try to predict: "this is yellow"

it will give me category A (or any other, because the probablity is the same for all categories in this case).

my question is: is there a classifier that would give me "unknown" (or none, or false, or error) for the test case above?

I would like to know when my test case could not be predicted with the given training set.

I think I could check if my_classifier.predict_proba(X_test)) returns an array with all equal or close values (in this example case: [[ 0.33333333 0.33333333 0.33333333]]).

actually, I would have to check if the values are close to their defaults, because the probabilities might not be the same for each category :)

so... any better approach or... is there a classifier with some confidence threshold I could use?

Christian Hirsch Christian Hirsch · Accepted Answer · 2016-05-10T03:47:53

If you have some unlabeled training data, you could add a dustbin class that contains all your unlabeled data. In your example this class would have the interpretation "not one of the colors green, blue or red". This approach is described in detail in http://arxiv.org/abs/1511.03719

scikit classifier with unknown prediction

2 Answers