Evaluate a prediction's accuracy

Question

I've trained and tested my KNN classifier. I've crossed validated it and the mean score is decent. Now, I would like to predict labels for some real data. But is there a way of seeing the accuracy of a prediction? I would like to actually save the predicted label only if the accuracy is high enough. I'm using Python and scikit-learn.

Have you checked the docs there are lots of ways to evaluate your model — EdChum
What do you mean by accuracy? Do you mean how many of the classifier's classifications match the real data (in which case you will need a point of truth to compare it to), or how certain the classifier is of each classification it makes? — jonrsharpe
@EdChum, I did, but that's model evaluation, not a prediction evaluation. — linkyndy
@jonrsharpe, I would like to know how certain is the classifier about the label it predicted. Say if prediction A has 90% accuracy, it's a good bet it's correct and I will save it to my database. — linkyndy
@AndreiHorak have you looked at scikit-learn.org/stable/modules/generated/…? — jonrsharpe

RPresle RPresle · Accepted Answer · 2015-04-14T10:15:18

As @jonrsharpe said, predict_proba should do it.

It provide an array with all the probabilities of the categories. Lets say, you have 3 categories [A, B, C]. The predict_proba method will return [0.2,0.3,0.5]. So here your accuracy would be

A=0.2
B=0.3
C=0.5

For example :

categories = [A, B, C]
X = # put your data
Y = # put your result

classifier.fit(X, Y)
prediction = classifier.predict_proba(X) # predict whatever you want here

for line in prediction:
    # numpy.argmax return the index of the biggest value in the array
    # max return the biggest value
    print("The class is %s with proba : %f " % (categories[numpy.argmax(line)], max(line, key=float)))

Important : take care of the order in your categories array. The content of the predict_proba results are sorted lexically based on the categories. Don't hesitate to sort categories before treating the result

Evaluate a prediction's accuracy

1 Answers