Use pROC package with h2o

Question

I'm doing a binary classification with a GBM using the h2o package. I want to assess the predictive power of a certain variable, and if I'm correct I can do so by comparing the AUC of a model with the specific variable and a model without the specific variable.

I'm taking the titanic dataset as an example.

So my hypothesis is: Age has significant predictive value whether someone will survive.

df <- h2o.importFile(path = "http://s3.amazonaws.com/h2o-public-test-data/smalldata/gbm_test/titanic.csv")
response <- "survived"
df[[response]] <- as.factor(df[[response]])           
## use all other columns (except for the name) as predictors
predictorsA <- setdiff(names(df), c(response, "name")) 
predictorsB <- setdiff(names(df), c(response, "name", "age")) 

splits <- h2o.splitFrame(
  data = df, 
  ratios = c(0.6,0.2),   ## only need to specify 2 fractions, the 3rd is implied
  destination_frames = c("train.hex", "valid.hex", "test.hex"), seed = 1234
)
train <- splits[[1]]
valid <- splits[[2]]
test  <- splits[[3]]

gbmA <- h2o.gbm(x = predictorsA, y = response, distribution="bernoulli", training_frame = train)
gbmB <- h2o.gbm(x = predictorsB, y = response, distribution="bernoulli", training_frame = train)

## Get the AUC
h2o.auc(h2o.performance(gbmA, newdata = valid))
[1] 0.9631624
h2o.auc(h2o.performance(gbmB, newdata = test))
[1] 0.9603211

I know the pROC package has a roc.test function to compare AUC of two ROC curves and I would like to apply this function on the outcomes of my h2o model.

Rushabh Patel Rushabh Patel · Accepted Answer · 2019-05-14T16:00:03

You can do something like this-

valid_A <- as.data.frame(h2o.predict(gbmA,valid))
valid_B <- as.data.frame(h2o.predict(gbmB,valid))
valid_df <- as.data.frame(valid)
roc1 <- roc(valid_df$survived,valid_A$p1)
roc2 <- roc(valid_df$survived,valid_B$p1)


> roc.test(roc1,roc2)

    DeLong's test for two correlated ROC curves

data:  roc1 and roc2
Z = -0.087489, p-value = 0.9303
alternative hypothesis: true difference in AUC is not equal to 0
sample estimates:
AUC of roc1 AUC of roc2 
  0.9500141   0.9504367

Use pROC package with h2o

1 Answers