Sklearn: GridSearchCV over n_iter parameter

Question

Sklearn recommends that for iterative estimators the number of iterations should be specified by the n_iter parameter of .fit().

Running a grid search for optimal hyperparameters with GridSearchCV allows you to specify only ranges of values for parameters that can be set with estimator.set_params().

So I am wondering, how do I run a grid search for (amongst others) the number of iterations? A solution would be to change the estimator code, so that n_iter is set in the constructor and by .set_params(), but for some reason this is explicitely not recommended, as explained above.

Edit: Here is an example:

Create an estimator that requires two parameters:

estimator = SomeEstimator(alpha=5, theta=0.001)

You can fit an estimator on data X and labels y with the fit() method. In addition, assume that the estimator runs an iterative algorithm and you can specify how many times it is supposed to run. Sklearn recommends to use the parameter n_iter of the fit() function for this:

estimator.fit(X, y, n_iter=4)

To find the optimal parameter values for alpha and theta you can run a GridSearchCV, which runs estimator.fit() for all combinations of those parameters:

GridSearchCV(estimator, {'alpha':[5, 6], 'theta':[0.1, 0.01, 0.001]}).fit(X, y, n_iter=4)

This will run the estimator for the combinations of alpha/theta:

(5, 0.1)
(5, 0.01)
(5, 0.001)
(6, 0.1)
(6, 0.01)
(6, 0.001)

But I want to these six combinations for different values of n_iter (1..10). Of course I can use a for-loop, but thats not the point of GridSearchCV.

piman314 piman314 · Accepted Answer · 2017-10-10T12:27:46

You can use GridSearchCV to find the best n_iter hyper parameter with an estimator that has that as a parameter, in the same way as other hyper parameters. The below works find on sklearn 0.19.0

np.random.seed(42)
clf = SGDClassifier()
params = {'n_iter': [30, 50, 100], 'alpha': [0.01, 0.1, 1.0]}
gs = GridSearchCV(clf, params, scoring='accuracy', cv=5)
X = np.random.random((100, 5))
y = np.random.choice([0, 1], 100)
gs.fit(X, y)
print gs.best_params_

Output:

{'alpha': 0.1, 'n_iter': 30}

Sklearn: GridSearchCV over n_iter parameter

1 Answers