Calibration prediction for multi-class classification

0
votes

How we can do Calibration prediction for multi-class classification?

I tried following https://machinelearningmastery.com/calibrated-classification-model-in-scikit-learn/ , but this doesn't work for multi-class problem as I get below error when i use sklearn.calibration.calibration_curve:

ValueError: Only binary classification is supported. Provided labels ['x' 'y' 'z' 'a' 'b'].

2
calibration

2 Answers

3
votes

One way to do it would be to retrain your multi-class model by using OneVsRestClassifier, then treat every class as a separate model. I've pasted a simple example down below that I used in a NLP project, hope it helps.

# Binarize the output
X_train = train_df['Text']
X_test = test_df['Text']

lb = LabelBinarizer()
y_train = lb.fit_transform(train_df['Target'])
y_test = lb.transform(test_df['Target'])

# Train a model with tfidf-vectorizer and LinearSVC
tfidf = TfidfVectorizer()
clf = LinearSVC()
clf = CalibratedClassifierCV(clf)
clf = OneVsRestClassifier(clf)

# Fit the model
pipe = Pipeline([('tfidf', tfidf), ('clf', clf)])
pipe.fit(X_train, y_train)

# Plot the Calibration Curve for every class
plt.figure(figsize=(20, 10))
ax1 = plt.subplot2grid((3, 1), (0, 0), rowspan=2)
ax2 = plt.subplot2grid((3, 1), (2, 0))
ax1.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")

targets = range(len(lb.classes_))
for target in targets:
    prob_pos = pipe.predict_proba(X_test)[:, target]
    fraction_of_positives, mean_predicted_value = calibration_curve(y_test[:, target], prob_pos, n_bins=10)
    name = lb.classes_[target]

    ax1.plot(mean_predicted_value, fraction_of_positives, "s-", label="%s" % (name, ))
    ax2.hist(prob_pos, range=(0, 1), bins=10, label=name, histtype="step", lw=2)

ax1.set_ylabel("The proportion of samples whose class is the positive class")
ax1.set_xlabel("The mean predicted probability in each bin")
ax1.set_ylim([-0.05, 1.05])
ax1.legend(loc="lower right")
ax1.set_title('Calibration plots (reliability curve)')

ax2.set_xlabel("Mean predicted value")
ax2.set_ylabel("Count")
ax2.legend(loc="upper center", ncol=2)

plt.tight_layout()
plt.show()
0
votes

The sklearn.calibration.calibration_curve gives you an error, because a calibration curve assumes inputs come from a binary classifier (see documentation).

However, the question you are asking is whether calibration is possible for multi-class classification problems. This is possible according to the scikit-learn documentation about calibration, it states:

CalibratedClassifierCV can also deal with classification tasks that involve more than two classes if the base estimator can do so. In this case, the classifier is calibrated first for each class separately in an one-vs-rest fashion. When predicting probabilities for unseen data, the calibrated probabilities for each class are predicted separately. As those probabilities do not necessarily sum to one, a postprocessing is performed to normalize them.