Always getting AUC = 0.5

Question

I'm trying to use a Convolutional Neural Network (CNN) to predict the classes of the test images, as follows:

for root, dirs, files in os.walk(test_directory):
    for file in files:
        img = cv2.imread(root + '/' + file)
        img = cv2.resize(img,(512,512),interpolation=cv2.INTER_AREA)
        img = np.expand_dims(img, axis=0)
        img = img/255.0
        if os.path.basename(root) == 'nevus':
            label = 1
        elif os.path.basename(root) == 'melanoma':
            label = 0
        labels.append(label)
        img_class = model.predict_classes(img)
        img_class_probability = model.predict(img)
        prediction_probability = img_class_probability[0]
        prediction_probabilities.append(prediction_probability)
        prediction = img_class[0]
        if prediction == label:
            correct_classification = correct_classification + 1
        number_of_test_images = number_of_test_images + 1

fpr, tpr, thresholds = roc_curve(labels, prediction_probabilities)
auc_value = auc(fpr, tpr)

I would like to ask why am I always having AUC = 0.5? Maybe I'm doing something wrong?

Thanks.

Have you calculated other metrics like accuracy, sensitivity, specificity and what scores did you get? — Suleiman
@Suleiman Thanks for your kind reply. I calculated the accuracy, which was around 77%. But, it seems that the classes predicted are all the same (i.e. always "melanoma"). — Simplicity
Your code doesn't appear to include the model. Please show us the code that constructs and trains the model (including the choice of learned parameter initializers and optimizer) — nanofarad
If your model is always predicting one class and you have a balanced dataset i.e same number of nevus and melanoma then that will explain why the AUC = 0.5 — Suleiman

FelixHo FelixHo · Accepted Answer · 2020-06-11T08:08:09

In this case, it may be that the thresholds of AUC are set incorrectly which resulting in all the model output values (score/probability) always be the same class under all threshold selections, so the AUC will always be 0.5

The following example illustrates the effect of thresholds on AUC

import tensorflow as tf # tf 2.0+

y_true = [0, 0, 1, 1, 1, 1]
y_pred = [0.001, 0.002, 0.003, 0.004, 0.005, 0.006]

#--------------------------------------------------------------
m1 = tf.keras.metrics.AUC(thresholds=[0.0, 0.5, 1.0])
m1.update_state(y_true, y_pred)

# when threshold=0.0 , y_class always be 1
# when threshold=0.5 , y_class always be 0
# when threshold=1.0 , y_class always be 0 

print('AUC={}'.format(m1.result().numpy())) 
# output: AUC=0.5


#--------------------------------------------------------------
m2 = tf.keras.metrics.AUC(thresholds=[0.0, 0.0045, 1.0])
m2.update_state(y_true, y_pred)

# when threshold=0.0    , y_class always be 1 
# when threshold=0.0045 , y_class will   be [0, 0, 0, 0, 1, 1]
# when threshold=1.0    , y_class always be 0 

print('AUC={}'.format(m2.result().numpy())) 
# output: AUC=0.75


#--------------------------------------------------------------
m3 = tf.keras.metrics.AUC(num_thresholds=300) 
# print(m3.thresholds)
m3.update_state(y_true, y_pred)

print('AUC={}'.format(m3.result().numpy())) 
# output: AUC=0.875

Always getting AUC = 0.5

1 Answers