How to pairwise compare two ROC curve using sklearn?

Question

I have to evaluate the performance of two classification algorithms. I obtain the False Positive Rate and the True Positive Rate using the roc_curve from sklearn (here documentation). I used the following code:

fpr = dict()
tpr = dict()
roc_auc = dict()
for i in range(2):
    fpr[i], tpr[i], _ = roc_curve(true_labels, pred_labels)
    roc_auc[i] = auc(fpr[i], tpr[i])

I have this information for two classifiers, and now I want to compare them. I know that it is possible to use AUC or other metrics, but I really want to understand the percentage of times a ROC curve is over the other. For this reason, I would like to compare pairwise points from the two curves.

My first idea was to compare, for each fpr element the corresponding tpr. Unfortunately, the fpr and tpr objects have different size, since they are the output of an interpolation step.

Any idea on how to do so? It seems that I cannot say to the roc_curve function how many (and which) points I want in the output, in order to make the comparison feasible. It is not mandatory to use sklearn.

Could you clarify what you mean by "the percentage of times a ROC curve is over the other"? Is the percentage meant as over the x-axis, the thresholds, something else? — Ben Reiniger
I mean that (assuming that I have 100k points) I want to know how many times the y-value of Curve1 is higher than Curve2 — A1010
Just to be sure: the 100k points are x-values? So you don't care how the thresholds between the two classifiers compare? — Ben Reiniger

OmG OmG · Accepted Answer · 2020-06-24T15:51:09

This solution is not specific to sklearn but is a scientific method. Using a Monte-Carlo method. Generate 10K points inside a unit square and count the number which is denoted by c, of points that is under both ROC curves. c/10000 is equal to the intersection area of these two curves as the area of a unit square is 1.

To generate a random point inside a unit square you can use the following code:

import random 

x = random.random()
y = random.random()

ِyou can check that the point of (x,y) is under a curve with the list of xs and ys by adding 0 to the tail of them and use of the following code:

is_under = True
for i in range(1, len(xs)): # 0 is added to the tails of xs and ys
    m = (y[1] - y[0])/(x[1]-x[0])
    if y > m * (x - x[0]) + y[0]:
        is_under = False
        break

In the above code, we checked that the given point (x,y) should be under any segment of the curve. If it is, at the end of the loop is_under must be True, and it must be False otherwise.

Notice that if you increase the number of points, the precision of the method will be increased.

How to pairwise compare two ROC curve using sklearn?

1 Answers