Using eml in caret: Error for class probabilities

Question

I am trying to compare the standard neural network approach to the extreme learning machine classifier (based on ROC metric), using the method "nnet" and "elm" in the R package caret. For nnet, everything works out, but using method = "elm" I get the following error(s):

Error in evalSummaryFunction(y, wts = weights, ctrl = trControl, lev = classLevels,  : 
  train()'s use of ROC codes requires class probabilities. See the classProbs option of trainControl()
In addition: Warning messages:
1: In train.default(x, y, weights = w, ...) :
  At least one of the class levels are not valid R variables names; This may cause errors if class probabilities are generated because the variables names will be converted to: X1, X2
2: In train.default(x, y, weights = w, ...) :
  Class probabilities were requested for a model that does not implement them

I also got the first error when method = "nnet", but here I could solve the problem by making score a factor variable. Hence, this cannot be the problem here.

I am relatively new to R, and maybe the error is trivial, but right now I am stuck... Since elmNN seems to be relatively newly implemented, I also couldn't find anything online on how to use elm in caret.

gc <- read.table("germanCreditNum.txt")
colnames(gc)[25]<-"score"

gc_inTrain <- createDataPartition(y = gc$score,
    ## the outcome data are needed
    p = .8,
    ## The percentage of data in the
    ## training set
    list = FALSE)

str(gc_inTrain)
gc_training <- gc[ gc_inTrain,]
gc_testing <- gc[-gc_inTrain,]
nrow(gc_training) ## No of rows 
nrow(gc_testing)

gc_training$score <- as.factor(gc_training$score)

gc_ctrl <- trainControl(method = "boot",
    repeats = 1,
    classProbs = TRUE,
    summaryFunction = twoClassSummary)

neuralnetFit <- train(score ~ .,
    data = gc_training,
    method = "nnet",
    trControl = gc_ctrl,
    metric = "ROC",
    preProc = c("center", "scale"))

neuralnetFit
plot(neuralnetFit)
nnClasses <- predict(neuralnetFit, newdata = gc_testing)
str(nnClasses)

## start with ELM for German Credit

gc_ctrl2 <- trainControl(classProbs = TRUE, summaryFunction = twoClassSummary)
elmFit <- train(score ~ ., 
    data = gc_training, 
    method = "elm", 
    trControl = gc_ctrl2, 
    metric = "ROC", 
    preProc = c("center", "scale"))

elmFit
plot(elmFit)

elmClasses <- predict(elmFit, newdata = gc_testing)
str(elmClasses)
elmProbs <- predict(elmFit, newdata = gc_testing, type = "prob")
head(elmProbs)

topepo topepo · Accepted Answer · 2014-11-16T02:39:52

I have no recollection why I didn't include a probability model for ELM (I probably had a good reason). You can use a custom method to get the softmax values:

library(caret)

set.seed(1)
dat <- twoClassSim(100)

elm_fun <- getModelInfo("elm")[[1]]
elm_fun$prob <- function (modelFit, newdata, submodels = NULL)  {
  out <- exp(predict(modelFit, newdata))
  t(apply(out, 1, function(x) x/sum(x)))
}
mod <- train(Class ~ ., data = dat, 
             method = elm_fun,
             metric = "ROC",
             trControl = trainControl(classProbs = TRUE,
                                      summaryFunction = twoClassSummary))

Max

Using eml in caret: Error for class probabilities

1 Answers