Calibrating AutoML models in H2O

Question

From what I can see in the docs, H2O supports calibration for GBM, DRF, XGBoost models only and has to be specified prior to the training phase.

I find it confusing. If calibration is a post-processing step and is model agnostic, shouldn't it be possible to calibrate any model trained using H2O, even after the training process is finished?

Currently, I'm dealing with a model that I've trained using AutoML. Even though it is a GBM model, I'm not able to easily calibrate it by providing a calibrate_model parameter as it is not supported by AutoML. I don't see any option to calibrate it after it's trained either.

Does anyone know an easy way to calibrate already-trained H2O models? Is it necessary to "manually" calibrate them using algorithms such as Platt scaling or is there a way to do it without using any extra libraries? Thanks

Answer 1

I find it confusing. If calibration is a post-processing step

The reason why it is part of model training right now is to have it in MOJO (our deployment artifact).

and is model agnostic, shouldn't it be possible to calibrate any model trained using H2O, even after the training process is finished?

Calibrating a model ex-post makes a lot of sense, all the code is already in - it “just” needs to be exposed to users. We created a ticket for this here.