automatically stop scipy.optimize.fmin_bfgs after n function calls (not BFGS iterations!)

Question

I use logistic regression with scipy.optimize.fmin_bfgs for minimizing the cost function. The cost function stays constant for my particular data set and BFGS does not converge, so I want to apply lasso+ridge regularization.

Now, I want to try out optimizing the cost for various values of the regularization parameters lambda1/2 in order to find the best combination:

for lambda1 in range(...):
    for lambda2 in range(..):
        scipy.optimize.fmin_bfgs(...) # Optimize cost with lambda1 and lambda2

The problem is that, because BFGS is not converging, it stays "forever" in the call for the first values of lambda1/2.

Is there a way to automatically stop fmin_bfgs after a while? The maxiter parameter does not help me, because I have 1000s of samples and a large number of features/sample, so it doesn't even finish one such iteration in acceptable time.

In scipy 0.11, fmin_bfgs has a maxfun parameter -- can one emulate this somehow in scipy 0.10?

EDIT: By popular demand, here are some relevant snippets of code:

The function computing the cost (the usual notations apply):

def computeCost(theta, X, y):
    h = sigmoid(X.dot(theta.T))
    J = y.T.dot(log(h)) + (1.0 - y.T).dot(log(1.0 - h))
    J_reg2 = theta[1:]**2
    J_reg1 = theta[1:]
    cost = (-1.0 / m) * (J.sum() + LAMBDA2 * J_reg2.sum() + LAMBDA1 * J_reg1.sum())
    return cost

Invoking the fmin_bfgs function:

initial_thetas = numpy.zeros((len(train_X[0]), 1))
myargs = (train_X, train_y)
theta = scipy.optimize.fmin_bfgs(computeCost, x0=initial_thetas, args=myargs)

A workaround way would be to add a sort of number of calls counter to the function itself and raise a special exception when the desired number of calls is reached. — ev-br
@Zhenya Thanks! I was hoping there might be an easier way, but I might end up doing that. — ACEG
@Denis What do you mean by "delta X"? I have my training samples in X, no modifications are done to them. — ACEG
sorry, theta: print theta, cost to see how they're moving, or deltas theta - theta prev, cost - costprev — denis
@Denis Aha, ok! Well, the problem is that the cost changes infinitesimally (around 8th decimal place or so), so it practically stays the same. I made a rather extensive post with my source code and outputs here on Cross-Validated.SE. — ACEG

Anake Anake · Accepted Answer · 2012-04-25T09:56:05

Your problem isn't the number of iterations. The reason why the lambda numbers aren't changing is that the optimisation isn't working. Scipy should be working out those numbers for you rather than you supplying them through for loops.

Maybe if you could include more code it would be easier to see how to fix it.

automatically stop scipy.optimize.fmin_bfgs after n function calls (not BFGS iterations!)

1 Answers