Best criteria for performance evaluation, error or accuracy?

Question

ANN and KNN on abalone data set using Weka.

Result for ANN Correctly Classified Instances 3183 76.203 % Incorrectly Classified Instances 994 23.797 % Mean absolute error 0.214 Root mean squared error 0.3349 Relative absolute error 58.6486 %

Result for KNN| Correctly Classified Instances 3211 76.8734 % Incorrectly Classified Instances 966 23.1266 % Mean absolute error 0.2142 Root mean squared error 0.3361 Relative absolute error 58.7113 %

KNN has high accuracy but ANN has low errors. So which of the two algorithms should I say is better? Which is more preferable criteria, accuracy or error?What I understood was that error should decrease with high accuracy. But the results here are opposite.Why is this so?

Answer 1

The answer depends on whether you want to treat the problem as classification (as suggested by the algorithms you use) or regression. If it's a classification problem, then you should only consider the % of correctly/incorrectly classified instances. Otherwise, the error.

To explain, the % of correctly classified instances takes only into account whether a prediction is accurate or not, i.e. predicting 2 instead of 1 is as incorrect as predicting 10000. The reasoning is that you get the class of the datum wrong and there is no notion of magnitude of difference between classes. For regression on the other hand, you predict a continuous quantity and the magnitude of difference matters. That is, if the actual value is 1 and the prediction 2, the model is much better than when the prediction is 10000.

This way you can get better accuracy with worse error or ice versa. What happens is that you get more correct predictions overall, but the ones that are wrong are off the mark further.

Which measure of performance you want to use really depends on your particular application. Do you simply care whether the correct class is predicted or not, or also about the distance to the correct prediction? If the latter is the case, I would recommend using regression instead of classification models.