Get precison model through GridSearchCV for recall optimization

Question

Given a machine learning model RBF SVC called 'm', I performed a gridSearchCV on gamma value, to optimize recall. I'm looking to answer to this: "The grid search should find the model that best optimizes for recall. How much better is the recall of this model than the precision?"

So I did the gridSearchCV:

grid_values = {'gamma': [0.001, 0.01, 0.05, 0.1, 1, 10, 100]}
grid_m_re = GridSearchCV(m, param_grid = grid_values, scoring = 'recall')
grid_m_re.fit(X_train, y_train)
y_decision_fn_scores_re = grid_m_re.decision_function(X_test) 

print('Grid best parameter (max. recall): ', grid_m_re.best_params_)
print('Grid best score (recall): ', grid_m_re.best_score_)

This tell me the best model is for gamma=0.001 and it has a recall score of 1.

I'm wondering how to get the precision for this model to get the trade of of this model, cause the GridSearchCV only has attribute to get what it was optimize for.([Doc sklearn.GridSearchCV][1])

Answer 1

Not sure if there's an easier/more direct way to get this, but this approach also allows you to capture the 'best' model to play around with later:

First do you CV fit on training data:

grid_m_re = GridSearchCV(m, param_grid = grid_values, scoring = 'recall')
grid_m_re.fit(X_train, y_train)

Once you're done, you can pull out the 'best' model (as determined by your scoring criteria during CV), and then use it however you want:

m_best = grid_m_re.best_estimator_

and in your specific case:

from sklearn.metrics import precision_score

y_pred = m_best.predict(X_test)
precision_score(y_test, y_pred)

Answer 2

You can easily overfit if you don't optimize both, C and gamma at the same time.

if you plot the SVC with C on the X axis, gamma on the y axis and recall as color you get some kind of V-Shape, see here

So if you do grid search, better optimize for both, C and gamma, at the same time.

The problem is that usually you get the best results for small C-Values, and in that area the V-shape has it's pointy end: is not very big and difficult to hit.

I recently used:

make a random grid of 10 points
      every point contains C, gamma, direction, speed
cut the dataset with stratifiedShuffleSplit
fit & estimate score with cross validation 

repeat:
  kill the worst two points
  the best two points spawn a kid
  move every point in its direction with just a little bit of random, 
  fit & estimate score with cross validation 
     (if a point notice it goes downward, turn around and half speed)
  until break criterion is hit

Worked like a charm.

I used the max distance in the feature space divided by four as initial speed, the direction had a maximum random of pi/4

Well, the cross validation was a bit costly.

Cleptocreatively inspired by this paper.

... and another edit:

I used between 10 and 20 cycles in the loop to get the perfect points. If your dataset is too big to do several fits, make a representative subset for the first few trainings...