Does this ROC curve make sense?

Question

This code returns and plots the true positive rate, false positive rate, true positive count, false positive count based on predicted and true values :

def get_all_stats(y_true , y_pred) : 

    def perf_measure(y_true, y_pred):

        TP = 0
        FP = 0
        TN = 0
        FN = 0

        for i in range(len(y_true)): 
            if y_true[i] == 1 and y_pred[i] == 1:
                TP += 1
            if y_pred[i]==1 and y_true[i]!=y_pred[i]:
                FP += 1
            if y_true[i]== 0 and y_pred[i]==0:
                TN += 1
            if y_pred[i]==0 and y_true[i] != y_pred[i]:
                FN += 1

        if(FP == 0) : 
            FPR = 0;
        else : 
            FPR = FP / (FP + TN)

        if(TP == 0) : 
            TPR = 0
        else : 
            TPR = TP / (TP + FN)

        return(TN , FPR, FN , TPR , TP , FP)

    tn, fpr, fn, tpr, tp , fp = perf_measure(y_true, y_pred)

    return tpr , fpr , tp , fp

tpr1 , fpr1 , tp1 , fp1 = get_all_stats(y_true=[1,1,1] , y_pred=[1,0,0])
tpr2 , fpr2 , tp2 , fp2 = get_all_stats(y_true=[1,0,1] , y_pred=[0,1,0])
tpr3 , fpr3 , tp3 , fp3 = get_all_stats(y_true=[0,0,0] , y_pred=[1,0,0])

plt.figure(figsize=(12,6))
plt.tick_params(labelsize=12)

print(tpr1 , fpr1 , tp1 , fp1)
print(tpr2 , fpr2 , tp2 , fp2)
print(tpr3 , fpr3 , tp3 , fp3)

plt.plot([fpr1,fpr2,fpr3], [tpr1 , tpr2, tpr3], color='blue', label='')
plt.ylabel("TPR",fontsize=16)
plt.xlabel("FPR",fontsize=16)
plt.legend()

The resultant ROC plot produced is :

In order to mimic three different false positive and true positive rates and different thresholds calculate these values by implementing the function get_all_stats three times with different

tpr1 , fpr1 , tp1 , fp1 = get_all_stats(y_true=[1,1,1] , y_pred=[1,0,0])
tpr2 , fpr2 , tp2 , fp2 = get_all_stats(y_true=[1,0,1] , y_pred=[0,1,0])
tpr3 , fpr3 , tp3 , fp3 = get_all_stats(y_true=[0,0,0] , y_pred=[1,0,0])

There are 9 instances to be classified to either 1 or 0 where the truth values are : [1,1,1,1,0,1,0,0,0]

At threshold1 the predicted values are [1,0,0] where the truth values at this threshold are [1,1,1].

At threshold2 the predicted values are [0,1,0] where the truth values at this threshold are [1,0,1].

At threshold3 the predicted values are [1,0,0] where the truth values at this threshold are [0,0,0].

As can see the generated plot of the classifier produced is different from 'typical' ROC curves :

As it first descends and then the false positive and true positive rates decrease causing the line to 'move back'. Have I implemented the ROC curve correctly ? Can the AUC be calculated for this curve ?

Answer 1

Okay, motivated to help because you have a lot of rep -> have helped a lot of others. Here we go.

This ROC curve does not make sense. The issue is that you are calculating the FPR/TPR on only subsets of your data at different thresholds. At each threshold, you should use all of the data to calculate FPR and TPR. Thus you seem to have 3 points in your plot, but you should only have had one point with the FPR/TPR for y_true = [1,1,1,1,0,1,0,0,0] and y_pred = [1,0,0,0,1,0,1,0,0]. To ensure that you have an actual ROC curve, though, you also can't just make up the y_pred values at the different thresholds - these need to come from actual predicted probabilities that are then thresholded appropriately. I modified your code a bit because I like to use numpy; here's how you could calculate an ROC curve.

# start with the true labels, as you did
y_true = np.array([1, 1, 1, 1, 0, 1, 0, 0, 0])
# and a predicted probability of each being a "1"
# I just used random numbers for these, but you would get them
# from your classifier
predictions = np.array([
    0.07485627, 0.72546085, 0.60287482,
    0.90537829, 0.75789236, 0.01852192,
    0.85425979, 0.36881312, 0.63893516
])

# now define a set of thresholds (the more thresholds, the better
# the curve will look). There's a smarter way to do this in practice
# (you can sort the predicted probabilities and just have one threshold
# between each), but this is just to help with understanding
thresholds = np.linspace(0, 1, 11) # 0.1, 0.2, ..., 1.0

fprs = []
tprs = []

# we can precompute which inputs are actually 1s/0s and how many of each
true_1_idx = np.where(y_true == 1)[0]
true_0_idx = np.where(y_true == 0)[0]
n_true_1 = len(true_1_idx)
n_true_0 = len(true_0_idx)

for threshold in thresholds:
    # now, for each threshold, we use that on the underlying probabilities
    # to get the actual predicted classes
    pred_classes = predictions >= threshold
    # and compute FPR/TPR from those
    tprs.append((pred_classes[true_1_idx] == 1).sum() / n_true_1)
    fprs.append((pred_classes[true_0_idx] == 1).sum() / n_true_0)

plt.figure(figsize=(12,6))
plt.tick_params(labelsize=12)

plt.plot(fprs, tprs, color='blue')
plt.ylabel("TPR",fontsize=16)
plt.xlabel("FPR",fontsize=16)

Note that an ROC curve is always non-decreasing in TPR (y-axis) as you increase in FPR (x-axis); that is, it goes up as you move to the right. This is clear from the how the thresholding works. At a threshold of 0, all predictions are "1", so we have FPR = TPR = 1. Increasing the threshold gives fewer predictions of "1", so FPR and TPR can only stay the same or decrease.

Note that, even if we used the optimal thresholds, there are still jumps in the curve because we have a finite amount of data, so a finite number of different TPR/FPR pairs we can get with any threshold. If you have enough data, though, then this begins to look smooth. Here I've replaced a few lines in the above code to get a smoother plot:

n_points = 1000
y_true = np.random.randint(0, 2, size=n_points)
predictions = np.random.random(n_points)

thresholds = np.linspace(0, 1, 1000)

If it's not clear, an AUC of 0.5 is the worst possible, and you can see that's what we get with random "predictions". If your AUC is worse than 0.5, you can flip every prediction to be better than 0.5 (and something is probably wrong with your model/training).

If you actually want to plot an ROC curve in practice, not just write it yourself to learn a little more, use sklearn's roc_curve. They also have roc_auc_score to get the AUC for you.