How to calculate AUC under twoClassSummary?

Question

here is my code:

train <- data.frame(***contain label, feature group 1 and feature group 2***)

formula <- label ~ features group 1

ctrl <- trainControl(method = "repeatedcv",
                     number = 10,
                     repeats = 5,
                     summaryFunction = twoClassSummary,
                     classProbs = T)

fit <- train(formula,
             data = train, 
             method = "glm", 
             metric = "ROC",
             trControl = ctrl,
             na.action = na.omit)

pred <- predict(fit, train)

my question is: How to calculate the AUC of pred?

I ve tried prSummary, ROCR and pROC, didn't work, it seems like that I can not calculate AUC when both of the obs and pred are exactly the same(levels-wise).

I m wondering if I can train with AUC as metric, how can I can't show the AUC?

p.s.

> levels(train$label)
[1] "classA" "classB"
> levels(as.factor(pred))
[1] "classA" "classB"

btw what I am doing is: fit multiple algo with caret and rank them by AUC, then I can choose the optimal one (base on AUC).

*reproducible example:

train set: iris

feature g1: first 2 features

feature g2: last 2 features

seed: 123*

Please provide a reproducible example. What didn't work, what did you try? — Calimo
This auc is based on the train set and is of little relevance in model selection. — missuse

recon recon · Accepted Answer · 2019-10-15T16:07:22

this one could be the possible answer but I am not so sure if it s right, tell me if i m wrong.

response = as.factor(as.numeric(train$label))

predictor = as.vector(as.numeric(pred))

library(pROC)

result = as.numeric(roc(response, predictor)$auc)

btw since pROC run very slow, can anyone help me to convert this under ROCR package? thx a lot :)

How to calculate AUC under twoClassSummary?

1 Answers