Scikit-learn default class

Question

I'm using scikit-learn to classify some texts, I have 10 classes and I'm using svm.SVC(kernel='linear',probability=True,class_weight='balanced'), MultinomialNB() and tree.DecisionTreeClassifier() to classify.

The tree classifiers are doing it great, the accuracy of the three are in the 80-90 percent (that's enough for my application).

The problem is that there are some text that do not belong to any of the 10 classes, I mean there should be an "Others" or "Not Class found" class, but i cannot found a way to do it, I have tried to add that class and train the classifiers with random texts but the result wasn't that good (50-60% accuracy).

I'm trying to use the probabilities that naive bayes gives me with the clf.predict_proba function and define a threshold but thats kind of "overfitting" the classifier.

Does any one have solved a problem like this? Thanks in advance.

NOTES:

I used 1000 text to train the classifiers

What do you mean by "there are some texts that do not belong to any of the 10 classes" ? You have them in your train set which is labeled right ? What's their label ? — MMF
@MMF is right, you can't just add random texts, if you are planning on classifying some texts as "others" , you have to include a representative sample of these into the training set — maxymoo
I would also add that you better have a training set for "others" which size reflects the real class distribution (so if you expect 10% of "Others", then the training set should contain 10% of Others as well). — Pascal Soucy
@MMF No, the random texts are in the Test Set, so I tested the classifiers with those texts and they have been clasified in one of the ten categories because I don't have an "Others" class. — user4143377
Your training data should be as close as possible to the data your classifier will have to classify. If you try to find 100 texts of each class, you are doing it the wrong way. Instead pick 1000 random (ideally more) documents you expect your classifier to have to classify, find a way to get their class labels (manual classification or using their predefined labels), and train your classifier with that. So maybe you will have 300 "others", or 30 or 2. A good sample will match the real distribution and this is important for prior estimations. — Pascal Soucy

Thomas Thomas · Accepted Answer · 2020-01-10T14:17:41

When faced with a similar problem I added empty data examples in the training set, labelled with the 'unknown' class. WHile some models do not perform well empty features, some models do (Bernoulli Naive Bayes, random forest...). Ensure then (by using necessary number of empty lines) the recall on the unknown class is 1. Your trained vectorizer will not recognize any input in new texts with nothing in common with the training data (all features at 0) and your model will then classify them unknown. Using pandas:

emptyline={ 'Text':'', 'label': 'Unknown'}
for i in range(300): 
    df=df.append(emptyline, ignore_index=True)

Scikit-learn default class

2 Answers