Using custom scoring function in sklearn

Question

Let's say I have the following custom loss function that I'm using in sci-kit learn. In this case I'm only scoring the observations where my model scores above 0.8.

def customLoss(y_true, y_pred):
    a = pd.DataFrame({'Actuals':y_true, 'Preds': y_pred})
    a = a.query('Preds > 0.8')
    return(precision_score(a['Actuals'], a['Preds']))

param_grid = {'C': [0.001, 0.01, 0.1, 1, 10]}
scorer = make_scorer(mf.customLoss ,greater_is_better = True)
grid = GridSearchCV(LogisticRegression(class_weight = 'balanced'), param_grid = param_grid, scoring = scorer, cv = 5)

However, let's say I wanted to make the threshold (0.8) configurable. Obviously I would need to add a third argument to my loss function like this:

def customLoss(y_true, y_pred, threshold):
        a = pd.DataFrame({'Actuals':y_true, 'Preds': y_pred})
        a = a.query('Preds > @threshold')
        return(precision_score(a['Actuals'], a['Preds']))

However, I'm a little confused on where in the make_scorer function I'd put this third argument?

MichaelD MichaelD · Accepted Answer · 2019-05-17T16:51:11

According to the docs make_scorer takes a **kwargs parameter that are passed on to the scoring function, so that you can add any additional parameters by name when you call the function. See the . Here's your code with the updated call to scorer

# New function with the `threshold` parameter
def customLoss(y_true, y_pred, threshold):
        a = pd.DataFrame({'Actuals':y_true, 'Preds': y_pred})
        a = a.query('Preds > @threshold')
        return(precision_score(a['Actuals'], a['Preds']))

...

 scorer = make_scorer(mf.customLoss ,greater_is_better = True, threshold = 0.8)

Using custom scoring function in sklearn

1 Answers