XGBoost model performance reporting on validation data

Question

I want to leverage the early.stop.round functionality of XGBoost to do non-overfit training. For this I use following code:

param2 <- list("objective" = "reg:linear",
                     "eval_metric" = "rmse",
                     "max_depth" = 15,
                     "eta" = 0.03,
                     "gamma" = 0,
                     "subsample" = 0.5,
                    "colsample_bytree" = 0.6,
                     "min_child_weight" = 5,
                     "alpha" = 0.15)

  watchlist <- list(train = xgb.DMatrix(data = train_matrix, label = output_train),
                  test = xgb.DMatrix(data = total_matrix[ind, ], label = as.matrix(output_total[ind, ])))

  bst <- xgboost(data=train_matrix, label=output_train, nrounds = 500, watchlist = watchlist,
                        early.stop.round=5,verbose = 2, param=param2, missing = NaN)

So as required I create train and test xgb.DMatrix for watchlist and pass it to xgboost(). I made sure verbose is there to print intermediate results. But with verbose=2 I get log like:

tree prunning end, 1 roots, 1692 extra nodes, 0 pruned nodes ,max_depth=15
[74]    train-rmse:0.129515
tree prunning end, 1 roots, 1874 extra nodes, 0 pruned nodes ,max_depth=15
[75]    train-rmse:0.128455
tree prunning end, 1 roots, 1826 extra nodes, 0 pruned nodes ,max_depth=15
[76]    train-rmse:0.127804
tree prunning end, 1 roots, 1462 extra nodes, 0 pruned nodes ,max_depth=15
[77]    train-rmse:0.126874
tree prunning end, 1 roots, 1848 extra nodes, 0 pruned nodes ,max_depth=15
[78]    train-rmse:0.125914

while with verbose=1 gives me:

[74]    train-rmse:0.129515
[75]    train-rmse:0.128455
[76]    train-rmse:0.127804
[77]    train-rmse:0.126874
[78]    train-rmse:0.125914

But none of this gives me model performance at each step for test DMatrix. I have also tried without success:

verbose=T and verbose=F.
changing name of test DMatrix to validation

What I am missing to get desired output.

abhiieor abhiieor · Accepted Answer · 2016-10-10T12:24:45

Apparently test dataset performance reporting could only be done using xgb.train() not with xgboost(). Relevant modified code (not copying param part above) looks like:

  dtrain <- xgb.DMatrix(data = train_matrix, label = output_train)
  dtest <- xgb.DMatrix(data = total_matrix[ind, ], label = as.matrix(output_total[ind, ]))
  watchlist <- list(train = dtrain, test = dtest)    
  bst <- xgb.train(data= dtrain, nrounds = 500, watchlist = watchlist,
                        prediction = T, early.stop.round=5,verbose = 1, param=param2, missing = NaN)

XGBoost model performance reporting on validation data

1 Answers