Leaderboard in H2O AutoML

Question

I have just started learn to use H2O Auto ML and I am trying out a binary Classification model.

I am trying to understand why do the rankings of the model change with every run.

The top 5 models remain in top 5, but the models slightly shift to a higher or lower rank.

While DRF was ranked 2nd once, the other time it raked 3rd.

There are couple of reasons I can speculate that causes changes.

1. Seed to the algorithm changes each time
2. There is no leader board frame assigned
3. RF involve random sampling as part of the process resulting in different trees built each time
4. The leader board will not change, some other change to data / code is responsible for the change.

Could you please help me understand this better.

Answer 1

It sounds like you're not setting a seed, so you should start there. In order for the algorithms with inherent randomness (e.g. XGBoost, GBM, Random Forest) to produce the same answer each time, a random seed must be set (at minimum). In H2O AutoML, there's a single seed argument (which gets piped down to all the individual algorithms) and if you set it to the same value each time, most of the models will be the same on repeated runs. By default, AutoML will also do cross-validation with random folds, so this also guarantees the same folds are used each time.

There are a few caveats -- H2O Deep Learning is not reproducible (by default) even if you set a seed, so those models will always change. Since the "All Models" Stacked Ensemble uses Deep Learning models in addition to a bunch of other models, the final ensemble will also be non-reproducible.

Lastly, you should use the max_models instead of max_runtime_secs to control how long AutoML should run for -- otherwise you may get a different number of models on the leaderboard (and in the All Models Stacked Ensemble) on subsequent runs.