XGBoost GPU out of memory

Question

I am trying to implement a grid search using XGBoost and the Hyperopt library. But I run into the problem shown in the figure: at the 213th configuration, an out of memory error appears. Since my dataset is not very large, I doubt that it is an overload problem due to the data and not even about the parameters of the model on which I grid search. This is because the previous configurations also have more features or training points but the training has not stalled.

terminate called after throwing an instance of 'thrust::system::system_error'
  what():  parallel_for failed: out of memory
Aborted (core dumped)

I think maybe it could be a problem related to an accumulation of GPU memory due to the various configurations tested and therefore it is necessary to release it from time to time. But I don't find anything about it.

Tell me yours, thank you.

I didn't find a direct solution. I used the workaround to reduce the number of hyperopt trials. If I did more trials, I restarted the hyperopt search using the checkpoint of the previous session @LuísCosta — piff
@LuísCosta Also in this case, I don't have a direct solution to suggest to you. But I can share this link link where you can find how you can continue reloading the hyperopt trials. — piff

nils-f nils-f · Accepted Answer · 2020-10-06T09:33:45

This is a known issue with XGBoost and related to the section 'Memory Usage' in https://xgboost.readthedocs.io/en/latest/gpu/index.html.

Hyperopt is causing a large loop of XGB instances keeping data in memory. You have to free the memory by serializing your model/booster or deleting it after predictions are carried out.

See this XGB issue for workaround and more info: https://github.com/dmlc/xgboost/issues/4668

XGBoost GPU out of memory

1 Answers