Understanding error performing cross-validation with xgboost in R

Question

I am running a cross validation with xgboost in R.

library(xgboost)

data(agaricus.train, package='xgboost')
data(agaricus.test, package='xgboost')
train <- agaricus.train
test <- agaricus.test

cv.res <- xgb.cv(data = train$data, label = train$label, nfold = 5,
                 nrounds = 2, objective = "binary:logistic")

The output is given below

[0]  train-error:0.000921+0.000343   test-error:0.001228+0.000687
[1]  train-error:0.001075+0.000172   test-error:0.001228+0.000687

I'm assuming the the error is simply the error average between all k-folds when performing regression and the mode when performing classification, is that correct? And what is the second term after the plus sign?

In regression, when computing the average between the k-folds, is it done considering equal weight for each fold or there are some cases where it's done giving more weight to specific folds?

Hack-R Hack-R · Accepted Answer · 2016-06-24T00:45:43

1. I'm assuming the the error is simply the error average between all k-folds when performing regression and the mode when performing classification

It's the error from the holdout crossfold (see the quote below which applies also to your 3rd question). By default it's the binary classification error rate for this objective function.

Actually, it's up to you which metric you want to see here. You can specify it with the metrics option. Your choices are:

error binary classification error rate
rmse Rooted mean square error
logloss negative log-likelihood function
auc Area under curve
merror Exact matching error, used to evaluate multi-class classification

When it is not specified, the evaluation metric is chosen according to objective function.

2. And what is the second term after the plus sign?

It's the standard deviation of the error estimate. It mentions it in the documentation. You can turn this on or off using showsd.

3. In regression, when computing the average between the k-folds, is it done considering equal weight for each fold or there are some cases where it's done giving more weight to specific folds?

According to the docs:

Of the nfold subsamples, a single subsample is retained as the validation data for testing the model, and the remaining nfold - 1 subsamples are used as training data.

Understanding error performing cross-validation with xgboost in R

1 Answers