0
votes

I am trying to get a AUC plot working using the AUC package in R. I am unsure of the error and new to this fit is the trained model: test is the test data

test$going_to_cross <- predict(fit, test, type="prob") 

prediction <- predict(fit, test, type="prob")
submit <- data.frame(cust_id = test$cust_id, already_crossed = test$flag_cross_over, predictions = prediction)
write.csv(submit, file = "../predictions /cross_sell_predictionsRF.csv", row.names = FALSE)

head(submit, 5)

print("predictions")
colnames(prediction) <- c("predictiona", "predictionb")
head(prediction)
which(submit$going_to_cross == 1)


print("names submit")
names(submit)

#predict_cross <- submit$going_to_cross.0
head(predict_cross, 5)

I get the output here as:

    cust_id already_crossed predictions.0   predictions.1
280 14080465    0           0.436   0.564
281 24047747    0           0.218   0.782 
282 10897483    0           0.606   0.394
283 14005276    0           0.448   0.552
284 18488402    0           0.284   0.716

[1] "predictions"

Out[317]:
    predictiona predictionb
280 0.436   0.564
281 0.218   0.782
282 0.606   0.394
283 0.448   0.552
284 0.284   0.716
285 0.104   0.896

The code from the package is:

auc(sensitivity(submit$predictions, submit$already_crossed))

And the warning message is:

Warning message: In is.na(x): is.na() applied to non-(list or vector) of type 'NULL'

Update:

# get the data into single vectors
 submit_pred <- matrix(submit$predictions.1)
 submit_cross <- matrix(submit$already_crossed)

 dt <- cbind(submit_pred, submit_cross)
  dt <- matrix(dt)


  names(dt) <- c("submit_pred", "submit_cross")

 roc_pred <- prediction(dt$submit_pred, dt$submit_cross)
 perf <- performance(roc_pred, "tpr", "fpr")
 plot(perf, col="red")
 abline(0,1,col="grey")

get area under the curve

performance(roc_pred,"auc")@y.values head(dt)

1
what does str(submit$predictions) and str(submit$already_crossed) return? are there any NA values in already_crossed? - bjoseph
I think the "predictions" column includes information from predicting successes and failures (1s and 0s). Just try to use "submit$predictionb" instead of "submit$predictions" in your last piece of code. - AntoniosK
You are passing two vectors as prediction and you have one vector for actual values when you do : auc(sensitivity(submit$predictions, submit$already_crossed)) and you break the model. Also, you have only 0 predictions and the ROC curve will not be obtained. I'll send you an example soon.... - AntoniosK

1 Answers

0
votes

Try to adjust this script to your dataset (Using package ROCR).

library(ROCR)

# example dataset with some 0 and some 1 values as actual observations
dt = data.frame(matrix(data=c(
14080465 ,  0 ,  0.436 , 0.564,
24047747 ,  1 ,  0.218 , 0.782 ,
10897483 ,  0 ,  0.606 , 0.394,
14005276 ,  0 ,  0.448 , 0.552,
18488402 ,  1 ,  0.284 , 0.716
), nrow = 5, ncol = 4, byrow = T))

names(dt) = c("cust_id", "already_crossed", "predictions.0",   "predictions.1")

# obtain ROC curve
roc_pred <- prediction(dt$predictions.1, dt$already_crossed)
perf <- performance(roc_pred, "tpr", "fpr")
plot(perf, col="red")
abline(0,1,col="grey")

# get area under the curve
performance(roc_pred,"auc")@y.values

You can also do it with your approach (Using package AUC):

library(AUC)

# example dataset with some 0 and some 1 values as actual observations
dt = data.frame(matrix(data=c(
14080465 ,  0 ,  0.436 , 0.564,
24047747 ,  1 ,  0.218 , 0.782 ,
10897483 ,  0 ,  0.606 , 0.394,
14005276 ,  0 ,  0.448 , 0.552,
18488402 ,  1 ,  0.284 , 0.716
), nrow = 5, ncol = 4, byrow = T))

names(dt) = c("cust_id", "already_crossed", "predictions.0",   "predictions.1")

auc(sensitivity(dt$predictions.1, as.factor(dt$already_crossed)))
plot(sensitivity(dt$predictions.1, as.factor(dt$already_crossed)))

As I've said before you have to pass only one vector of predictions. Also, you need to save the actual classes (0s and 1s) as factors, otherwise the sensitivity function will break. However, I think what you want to compute (using your method) is this :

auc(roc(dt$predictions.1, as.factor(dt$already_crossed)))
plot(roc(dt$predictions.1, as.factor(dt$already_crossed)))