mlr: Filter Methods with Tuning

Question

This section of the ml tutorial: https://mlr.mlr-org.com/articles/tutorial/nested_resampling.html#filter-methods-with-tuning explains how to use a TuneWrapper with a FilterWrapper to tune the threshold for the filter. But what if my filter has hyperparameters that need tuning as well, such as a random forest variable importance filter? I don't seem to be able to tune any parameters except the threshold.

For example:

library(survival)
library(mlr)

data(veteran)
set.seed(24601)
task_id = "MAS"
mas.task <- makeSurvTask(id = task_id, data = veteran, target = c("time", "status"))
mas.task <- createDummyFeatures(mas.task)
tuning = makeResampleDesc("CV", iters=5, stratify=TRUE)                             # Tuning: 5-fold CV, no repeats

cox.filt.rsfrc.lrn = makeTuneWrapper(
      makeFilterWrapper(
        makeLearner(cl="surv.coxph", id = "cox.filt.rfsrc", predict.type="response"), 
        fw.method="randomForestSRC_importance",
        cache=TRUE,
        ntree=2000
      ), 
      resampling = tuning, 
      par.set = makeParamSet(
          makeIntegerParam("fw.abs", lower=2, upper=10),
          makeIntegerParam("mtry", lower = 5, upper = 15),
          makeIntegerParam("nodesize", lower=3, upper=25)
      ), 
      control = makeTuneControlRandom(maxit=20),
      show.info = TRUE)

produces the error message:

Error in checkTunerParset(learner, par.set, measures, control) : Can only tune parameters for which learner parameters exist: mtry,nodesize

Is there any way to tune the hyperparameters of the random forest?

EDIT: Other attempts following suggestion in comments:

Wrap tuner around base learner before feeding to filter (filter not shown) - fails

cox.lrn =  makeLearner(cl="surv.coxph", id = "cox.filt.rfsrc", predict.type="response")
cox.tune = makeTuneWrapper(cox.lrn, 
                   resampling = tuning, 
                   measures=list(cindex),
                   par.set = makeParamSet(
                     makeIntegerParam("mtry", lower = 5, upper = 15),
                     makeIntegerParam("nodesize", lower=3, upper=25),
                     makeIntegerParam("fw.abs", lower=2, upper=10)
                   ),
                   control = makeTuneControlRandom(maxit=20),
                   show.info = TRUE)

Error in checkTunerParset(learner, par.set, measures, control) : 
Can only tune parameters for which learner parameters exist: mtry,nodesize,fw.abs

Two levels of tuning - fails

cox.lrn =  makeLearner(cl="surv.coxph", id = "cox.filt.rfsrc", predict.type="response")
cox.filt = makeFilterWrapper(cox.lrn,
                     fw.method="randomForestSRC_importance",
                     cache=TRUE,
                     ntree=2000)
cox.tune = makeTuneWrapper(cox.filt, 
                   resampling = tuning, 
                   measures=list(cindex),
                   par.set = makeParamSet(
                     makeIntegerParam("fw.abs", lower=2, upper=10)
                   ),
                   control = makeTuneControlRandom(maxit=20),
                   show.info = TRUE)

cox.tune2 = makeTuneWrapper(cox.tune, 
                   resampling = tuning, 
                   measures=list(cindex),
                   par.set = makeParamSet(
                     makeIntegerParam("mtry", lower = 5, upper = 15),
                     makeIntegerParam("nodesize", lower=3, upper=25)
                   ),
                   control = makeTuneControlRandom(maxit=20),
                   show.info = TRUE)

Error in makeBaseWrapper(id, learner$type, learner, learner.subclass = c(learner.subclass,  : 
  Cannot wrap a tuning wrapper around another optimization wrapper!

Have you tried wrapping the learner with a tune wrapper which goes into the filter wrapper? — pat-s
@pat-s I cannot get this to work, but it is something that is really necessary. I have edited the question to show other things I have tried. — panda

pat-s pat-s · Accepted Answer · 2019-09-19T07:47:40

It looks like that you currently can not tune hyperparameters of filters. You can manually change certain parameters by passing them in makeFilterWrapper() but not tune them. You can only tune one of fw.abs, fw.perc or fw.tresh when it comes to filtering.

I do not know how big the effect will be on the ranking when using different hyperpars for the RandomForest filter. One way to check the robustness would be to compare the rankings of single RF model fits using different settings for mtry and friends with the help of getFeatureImportance(). If there is a very high rank correlation between these, you can safely ignore the tuning of the RF filter. (Maybe you want to use a different filter which does not come with this issue at all?)

If you insist on having this feature, you might need to raise PR for the package :)

lrn = makeLearner(cl = "surv.coxph", id = "cox.filt.rfsrc", predict.type = "response")

filter_wrapper = makeFilterWrapper(
  lrn,
  fw.method = "randomForestSRC_importance",
  cache = TRUE,
  ntrees = 2000
)

cox.filt.rsfrc.lrn = makeTuneWrapper(
  filter_wrapper,
  resampling = tuning,
  par.set = makeParamSet(
    makeIntegerParam("fw.abs", lower = 2, upper = 10)
  ),
  control = makeTuneControlRandom(maxit = 20),
  show.info = TRUE)

mlr: Filter Methods with Tuning

1 Answers