when does xgboost cross validation stop

Question

param = {'max_depth': 2, 'eta': 1, 'silent': 0, 'objective': 
         'multi:softmax', 'num_class': 10}
num_round = 1
res = xgb.cv(param, dtrain, num_round, nfold=10,
           metrics={'merror'}, seed=0, verbose_eval=True,
           callbacks=[xgb.callback.print_evaluation(show_stdv=True),
                      xgb.callback.early_stop(3)])

I can see a lot of the following logs:

[17:50:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 4 extra nodes, 0 pruned nodes, max_depth=2
[17:50:22] src/tree/updater_prune.cc:74: tree pruning end, 1 roots, 6 extra nodes, 0 pruned nodes, max_depth=2
......

Finally, I print out res as following:

[0] train-merror:0.800139+0.00308927    test-merror:0.815893+0.0139572

My question is that:

1, What does the train-merror:0.800139 and test-merror:0.815893 mean? Is it the mean value of 10 folder's eval data?

2, When do we need to set num_round > 1? I did a misunderstanding about the num_round between cv() and train(). When cv(), num_round just do another 10 folder cross validation again. But when train(), num_round set the tree counts I want. Is that right?

3, When cv(), in one iteration, when does the process end if I do not set early stoping?

4, How can I print some metric when a folder process end?

Thank you!

About question 2, I have realized that it was a stupid question. Both num_round in cv() and train() are the same meaning. And question3 is also be cancelled by question2 — iloveml

T. Scharf T. Scharf · Accepted Answer · 2017-07-02T00:02:10

1, What does the train-merror:0.800139 and test-merror:0.815893 mean? Is it the mean value of 10 folder's eval data?

[0] train-merror:0.800139+0.00308927  test-merror:0.815893+0.0139572

The first column is the average training error for ALL of the 10 folds for that round, and the 2nd column is the average 'out of fold/test' error for ALL of the folds for that round -- it also has the standard deviation there .. appended

regarding 2 & 3 ... num_round is the maximum number of boosting rounds for training. Training will cease if you build num_round trees before early stopping is engaged. Set it high if you are using early stopping, and let early stopping decide when to stop.

Finally -- early stopping is an integer that tells you how many rounds to still continue training, even if your test error NO LONGER IMPROVES -- this allows for some wiggle room as different rounds can be noisy w.r.t error.

If you have ever looked at training curves sometimes they get worse, then get better. This allows for some leniency in that area so you dont stop training too early. Set this at probably a minimum of 10, xgboost will save the actual best round. Plus if your ETA is small enough it'll be an inconsequential number. Set your ETA low enough to get at least a few hundred rounds IMO.

The res object will have all the relevant information from training. Just save that object or print some info from it...

when does xgboost cross validation stop

1 Answers