We have trained an Extra Tree model for some regression task. Our model consists of 3 extra trees, each having 200 trees of depth 30. On top of the 3 extra trees, we use a ridge regression. We train our model for several hours and pickle the trained model (the entire class object), for later use. However, the size of saved trained model is too big, about 140 GB! Is there a way to reduce the size of the saved model? are there any configuration in pickle that could be helpful, or any alternative for pickle?
2 Answers
4
votes
19
votes
You can try using joblib with compression parameter.
from sklearn.externals import joblib
joblib.dump(your_algo, 'pickle_file_name.pkl',compress=3)
compress - from 0 to 9. Higher value means more compression, but also slower read and write times. Using a value of 3 is often a good compromise.
You can use python standard compression modules zlib, gzip, bz2, lzma and xz. To use that you can just specify the format with specific extension
example
joblib.dump(obj, 'your_filename.pkl.z') # zlib
More information, see the [link]:(http://gael-varoquaux.info/programming/new_low-overhead_persistence_in_joblib_for_big_data.html)
joblibcan be more efficient in some cases. - Tchotchke