Random Forest in R: Difference between reported and observed error values

Question

I tried fitting a Random Forest on a dataset. It took hours but eventually fitted. Command used was: model <- train (classe~., data=training, method="rf", prox=F)

model reported following:

#

Random Forest

13737 samples 52 predictor 5 classes: 'A', 'B', 'C', 'D', 'E'

No pre-processing Resampling: Bootstrapped (25 reps)

Summary of sample sizes: 13737, 13737, 13737, 13737, 13737, 13737, ...

Resampling results across tuning parameters:

mtry   Accuracy    Kappa       Accuracy SD   Kappa SD

2    0.9888012  0.9858367  0.001844763  0.002329859
27   0.9882821  0.9851812  0.001874991  0.002365894
52   0.9820495  0.9773000  0.003680805  0.004649905

Accuracy was used to select the optimal model using the largest value. The final value used for the model was mtry = 2.

#

When I ran the prediction on the training data, I got 100% accuracy.

prediction <- predict(model, training)

table(prediction, training$classe)

prediction A B C D E

        A 3906    0    0    0    0
        B    0 2658    0    0    0
        C    0    0 2396    0    0
        D    0    0    0 2252    0
        E    0    0    0    0 2525

I would've expected it to be ~98.9% as reported by the model.

What am I missing ?

Thanks and regards,

Ozan147 Ozan147 · Accepted Answer · 2018-08-05T00:48:19

The same answer from this question applies: https://stackoverflow.com/a/51690010/9684157

You can interpret these values as in-sample accuracies with and without resampling, respectively.

The package caret performs bootstrapped resampling with 25 repetition when you fit the model, which can be seen in your model output. So, the accuracy value is based on 25 x 13737 observations. In order to create the confusion matrix, you are using the final model (the one with mtry = 2) to predict the outcomes from the training sample, which has a length of 13737. Therefore, it is normal to have a slight difference in corresponding accuracies.

You need to be careful while assessing the goodness-of-fit because you are training and evaluating your model using the same dataset. No surprise that you get a large accuracy. It is always good to evaluate your final model with an unseen dataset to ensure its performance and detect possible over-fitting issues.

Random Forest in R: Difference between reported and observed error values

1 Answers