Logistic Regression and Scipy Optimization with fmin_bfgs

Question

I am trying to implement Logistic Regression and I am using Scipy's Optimize module to find the optimized theta values. I am able to get the correct value when using the fmin function. But I would like to do so when using the fmin_bfgs function which requires the gradient.

Here is a code snippet:

#Returns Cost of current theta values and gradient of cost w.r.t theta.
def costFunction(theta, X, y, _lambda):

    #Initializes useful variables
    m = len(y)
    grad = np.zeros((np.shape(theta)))

    #Solves for Hypothesis for input X.
    h_x = Sigmoid(np.dot(X,theta))

    #Reshaped because numpy kept returning row vector, not column.
    h_x = h_x.reshape((m,1))

    #Cost is broken up into terms for simplicity. 
    term1 = -y*np.log(h_x)
    term2 = (1-y)*np.log((1-h_x))



    #Regularized Cost FUnction
    J = (1/m) * sum(term1-term2) + (_lambda/(2*m)) * sum(np.square(theta[1:][:]))

    #Gradient for Theta1
    grad_reg = (_lambda/m)*theta[1:]

    #Combines gradient for Theta1 and onward. 
    grad = (1/m)* (np.dot(np.transpose(X),(h_x-y))) + np.vstack(([0],grad_reg)) 

    return J,grad 





#Finds Optimal Value for theta
cost, grad = costFunction(initial_theta, X,y, _lambda)
opt_theta = fmin_bfgs(cost,x0=initial_theta,fprime=grad, args = (X,y,_lambda))

The error I get is 'numpy.ndarray' object is not callablewhich comes from the function_wrapper function in the optimize module. I even tried returning the gradient and the cost in two different functions, but got a vstack error of some sort(if that matters/helps at all).

As far as I can see I have provided what the optimization function ask for.

EDIT/UPDATE: I realized that the error I was getting was because I was passing the cost, and grad numpy arrays as parameters when it was expecting a function to return those parameters. I realize that I could create a wrapper function? to get both of those values without using two separate functions, but for temp purposes, I changed costFunction so it only returns the cost, and created a whole new function,Grad(), (with identical code though) that only returns the grad. This gives me the all the input array dimensions except for the concatenation axis must match exactly vstack error again.

ev-br ev-br · Accepted Answer · 2014-12-03T12:29:09

It's hard to debug things without a minimal reproducible example.

The way I'd debug it, I'd start with something trivially simple to make sure I get the basic syntax right. There are several ways of using bfgs minimzation with an explicit gradient. First, no gradient information:

In [1]: import numpy as np

In [2]: from scipy.optimize import minimize

In [3]: def f(x):
   ...:     return np.sum((x-2.)**2)
   ...: 

In [4]: x0 = np.ones(3)

In [5]: minimize(f, x0, method='bfgs')
Out[5]: 
  status: 0
 success: True
    njev: 4
    nfev: 20
     fun: 1.6656677750444977e-16
       x: array([ 1.99999999,  1.99999999,  1.99999999])
<snip>

Now, with gradient, you can either have a callable which returns the function and the gradient:

In [6]: def f_and_jac(x):
   ...:     val = np.sum((x-2.)**2)
   ...:     grad = 2.*(x-2.)
   ...:     return val, grad
   ...: 

In [7]: minimize(f_and_jac, x0, method='bfgs', jac=True)   # notice the 'jac' parameter
Out[7]: 
  status: 0
 success: True
    njev: 4
    nfev: 4
     fun: 0.0
       x: array([ 2.,  2.,  2.])

Alternatively, you can set jac to a callable, which should have the same signature as the cost function and return the gradient:

In [8]: def jac(x):
   ...:     return 2.*(x-2.)
   ...: 

In [9]: minimize(f, x0, method='bfgs', jac=jac)
Out[9]: 
  status: 0
 success: True
    njev: 4
    nfev: 4
     fun: 0.0
       x: array([ 2.,  2.,  2.])

Logistic Regression and Scipy Optimization with fmin_bfgs

1 Answers