mlr3 predictions to new data with parameters from autotune

Question

I have a follow-up question to this one. As in the initial question, I am using the mlr3verse, have a new dataset, and would like to make predictions using parameters that performed well during autotuning. The answer to that question says to use at$train(task). This seems to initiate tuning again. Does it take advantage of the nested resampling at all by using those parameters?

Also, looking at at$tuning_result there are two sets of parameters, one called tune_x and one called params. What is the difference between these?

Thanks.

Edit: example workflow added below

library(mlr3verse)

set.seed(56624)
task = tsk("mtcars")

learner = lrn("regr.xgboost")

tune_ps = ParamSet$new(list(
  ParamDbl$new("eta", lower = .1, upper = .4),
  ParamInt$new("max_depth", lower = 2, upper = 4)
))

at = AutoTuner$new(learner = learner, 
                   resampling = rsmp("holdout"), # inner resampling
                   measures = msr("regr.mse"), 
                   tune_ps = tune_ps,
                   terminator = term("evals", n_evals = 3),
                   tuner = tnr("random_search"))

rr = resample(task = task, learner = at, resampling = rsmp("cv", folds = 2),
              store_models = TRUE)
rr$aggregate()
rr$score()
lapply(rr$learners, function(x) x$tuning_result)

at$train(task)
at$tuning_result

notreallynew.df = as.data.table(task)
at$predict_newdata(newdata = notreallynew.df)

This doesn't look like a coding question; more like a theory Q. — IRTFM
I mean it as a coding/R6 objects question, not a theory question. After running nested resampling to tune a model (via resample or benchmark), one has to train the model using at$train(task) to be able to generate a prediction. Does that training incorporate anything learned during the nested resampling? If not, what code can be used to take parameters from the nested resampling, train the model using those, and then generate predictions to a novel dataset? — Helen S
Not enough data or code to support an effort at an answer. You need to post enough to a) show what you have tried (with code and data example) and b) show what efforts at searching for an answer. — IRTFM
All of this should be explained in mlr3book.mlr-org.com or the help pages of the functions you used. If not, please open an issue in the respective repo. — pat-s
I've added an example. I did go through the book, and I've looked through some of the use cases in the gallery. If there's an example there with prediction to new data after nested resampling, please do point me to it since I missed it. — Helen S

pat-s pat-s · Accepted Answer · 2020-05-07T10:30:39

As ?AutoTuner tells, this class fits a model with the best hyperparameters found during the tuning. This model is then used for prediction, in your case to newdata when calling its method .$predict_newdata().

Also in ?AutoTuner you see the documentation linked to ?TuningInstance. This then tells you what the $tune_x and params slots represent. Try to look up the help pages next time - that's what they are there for ;)

This seems to initiate tuning again.

Why again? It does it in the first place, on all observations of task. I assume you might confuse yourself by the common misconception between "train/predict" vs. "resample". Read more about the theoretical differences of both to understand what both are doing. They have completely different aims and are not connected. Maybe the following reprex makes it more clear.

library(mlr3verse)
#> Loading required package: mlr3
#> Loading required package: mlr3db
#> Loading required package: mlr3filters
#> Loading required package: mlr3learners
#> Loading required package: mlr3pipelines
#> Loading required package: mlr3tuning
#> Loading required package: mlr3viz
#> Loading required package: paradox

set.seed(56624)
lgr::get_logger("mlr3")$set_threshold("warn")
task = tsk("mtcars")
learner = lrn("regr.xgboost")

tune_ps = ParamSet$new(list(
  ParamDbl$new("eta", lower = .1, upper = .4),
  ParamInt$new("max_depth", lower = 2, upper = 4)
))

at = AutoTuner$new(
  learner = learner,
  resampling = rsmp("holdout"), # inner resampling
  measures = msr("regr.mse"),
  tune_ps = tune_ps,
  terminator = term("evals", n_evals = 3),
  tuner = tnr("random_search"))

# train/predict with AutoTuner
at$train(task)
notreallynew.df = as.data.table(task)
at$predict_newdata(newdata = notreallynew.df)
#> <PredictionRegr> for 32 observations:
#>     row_id truth response
#>          1  21.0 9.272631
#>          2  21.0 9.272631
#>          3  22.8 9.272631
#> ---                      
#>         30  19.7 9.272631
#>         31  15.0 5.875841
#>         32  21.4 9.272631

# resample with AutoTuner for performance estimation

rr = resample(
  task = task, learner = at, resampling = rsmp("cv", folds = 2),
  store_models = TRUE)
rr$aggregate()
#> regr.mse 
#> 240.5866
rr$score()
#>          task task_id     learner         learner_id     resampling
#> 1: <TaskRegr>  mtcars <AutoTuner> regr.xgboost.tuned <ResamplingCV>
#> 2: <TaskRegr>  mtcars <AutoTuner> regr.xgboost.tuned <ResamplingCV>
#>    resampling_id iteration prediction regr.mse
#> 1:            cv         1     <list> 220.8076
#> 2:            cv         2     <list> 260.3656

^{Created on 2020-05-07 by the reprex package (v0.3.0)}

mlr3 predictions to new data with parameters from autotune

1 Answers