Adding metrics to default train() output from the caret package

Question

I would like to add additional metrics other than RMSE and Rsquared to the output of my linear model that I creating with the caret package. From what I understand, the code below will output the repeated cross-validated RMSE and Rsquared:

library(caret)
lm_reg1 <- train(log1p(mpg) ~ log1p(hp) + log1p(disp),
                 data = mtcars,
                 trControl = trainControl(method = "repeatedcv",
                                          number = 10,
                                          repeats = 10),
                 method = 'lm')
lm_reg

Output:

Linear Regression 

32 samples
10 predictors

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 10 times) 
Summary of sample sizes: 30, 29, 28, 29, 29, 28, ... 
Resampling results:

  RMSE       Rsquared 
  0.1134972  0.8808378

I know I can modify the output to a custom metric by modifying the summaryFunction in trainControl and referring to it's name in the metric parameter. Here's an example of one that I created that calculates the MAPE of a log-log model:

mape <- function(actual, predicted){
  mean(abs((actual - predicted)/actual))
}
mapeexpSummary <- function (data,
                            lev = NULL,
                            model = NULL) {
  out <- mape(expm1(data$obs), expm1(data$pred))  
  names(out) <- "MAPEEXP"
  out
}
lm_reg2 <- train(log1p(mpg) ~ log1p(hp) + log1p(disp),
                data = mtcars,
                trControl = trainControl(method = "repeatedcv",
                                         number = 10,
                                         summaryFunction = mapeexpSummary,
                                         repeats = 10),
                metric = 'MAPEEXP',
                method = 'lm')
lm_reg2

Output:

Linear Regression 

32 samples
10 predictors

No pre-processing
Resampling: Cross-Validated (10 fold, repeated 10 times) 
Summary of sample sizes: 28, 29, 29, 28, 28, 30, ... 
Resampling results:

  MAPEEXP  
  0.1022028

Is there any way to add them to a single output? I'm looking to save all of these values, but want to avoid creating two identical models to do so.

chinsoon12 chinsoon12 · Accepted Answer · 2016-08-18T03:47:07

add in RMSE and Rsquared in your mapeexpSummary?

mapeexpSummary <- function (data,
    lev = NULL,
    model = NULL) {
    c(MAPEEXP=mape(expm1(data$obs), expm1(data$pred)),
        RMSE=sqrt(mean((data$obs-data$pred)^2)),
        Rsquared=summary(lm(pred ~ obs, data))$r.squared)
}

Adding metrics to default train() output from the caret package

1 Answers