How to fix Value error reported when using optimize.minimize() but the function referenced has no error?

Question

To get the optimal value of Theta in logistic regression I was using optimize.minimize() function, and my function costFunction(X,y,theta) returns the cost and gradient given the value of X,y and theta. I've checked my function costFunction() with an initial value of theta and it's working fine. But on referencing this function in optimize.minimize() it reports value error.

Here are my codes for costFunction and where i call the function optimize.minimize()

def costFunction(X,y,theta):
    J = 0.0
    m = Y.size
    J = -1/m * np.sum(((1-y)*np.log(1-sigmoid(np.dot(X,theta))))+((y)*np.log(sigmoid(np.dot(X,theta)))))
    grad = 1/m*np.dot(X.T,(sigmoid(np.dot(X,theta))-y))
    return J, grad ```

#To check the function :
print(X[:,:3].shape)
J,grad = costFunction(X[:,:3],Y,theta=[0,0,0])
print(J)
print( grad)

#and this returns the following output:
(1000, 3)
0.6931471805599454
[ 0.      17.25682  5.92721]

#and here's where I call optimize.minimize() function:
options = {'maxiter' : 400}
initial_theta = np.zeros(3)
x = X[:,:3]
#res = optimize.minimize(computeCost,initial_theta,(X[:,:3],Y),jac = True,method = 'TNC',options = options)
res = optimize.minimize(costFunction,
                        initial_theta,
                        (x, Y),
                        jac=True,
                        method='TNC',
                        options=options)

cost = res.fun
theta = res.x
print("cost ".cost)
print("theta ".theta)
#and it returns the following error :

ValueError                                Traceback (most recent call last)
<ipython-input-69-55576d96c00a> in <module>
      8                         jac=True,
      9                         method='TNC',
---> 10                         options=options)
     11 
     12 cost = res.fun

~/anaconda3/lib/python3.7/site-packages/scipy/optimize/_minimize.py in minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints, tol, callback, options)
    604     elif meth == 'tnc':
    605         return _minimize_tnc(fun, x0, args, jac, bounds, callback=callback,
--> 606                              **options)
    607     elif meth == 'cobyla':
    608         return _minimize_cobyla(fun, x0, args, constraints, **options)

~/anaconda3/lib/python3.7/site-packages/scipy/optimize/tnc.py in _minimize_tnc(fun, x0, args, jac, bounds, eps, scale, offset, mesg_num, maxCGit, maxiter, eta, stepmx, accuracy, minfev, ftol, xtol, gtol, rescale, disp, callback, **unknown_options)
    407                                         offset, messages, maxCGit, maxfun,
    408                                         eta, stepmx, accuracy, fmin, ftol,
--> 409                                         xtol, pgtol, rescale, callback)
    410 
    411     funv, jacv = func_and_grad(x)

~/anaconda3/lib/python3.7/site-packages/scipy/optimize/tnc.py in func_and_grad(x)
    369     else:
    370         def func_and_grad(x):
--> 371             f = fun(x, *args)
    372             g = jac(x, *args)
    373             return f, g

~/anaconda3/lib/python3.7/site-packages/scipy/optimize/optimize.py in __call__(self, x, *args)
     61     def __call__(self, x, *args):
     62         self.x = numpy.asarray(x).copy()
---> 63         fg = self.fun(x, *args)
     64         self.jac = fg[1]
     65         return fg[0]

<ipython-input-65-97115ec06e6e> in costFunction(X, y, theta)
      2     J = 0.0
      3     m = Y.size
----> 4     J = -1/m * np.sum(((1-y)*np.log(1-sigmoid(np.dot(X,theta))))+((y)*np.log(sigmoid(np.dot(X,theta)))))
      5     grad = 1/m*np.dot(X.T,(sigmoid(np.dot(X,theta))-y))
      6     return J, grad

ValueError: shapes (3,) and (1000,) not aligned: 3 (dim 0) != 1000 (dim 0)```

Can you try breaking up the J computation line so that we know which operation is causing the error? — rdas

ml4294 ml4294 · Accepted Answer · 2019-04-01T18:39:25

It seems that the error is caused by the order of the arguments in the call of optimizer.minimize():

def costFunction(X,y,theta):
    J = 0.0
    m = y.size
    print(y.shape)
    print(X.shape)
    print(theta.shape)
    J = -1/m * np.sum(((1-y)*np.log(1-sigmoid(np.dot(X,theta))))+((y)*np.log(sigmoid(np.dot(X,theta)))))
    grad = 1/m*np.dot(X.T, (sigmoid(np.dot(X, theta))-y))
    return J, grad

This will print different output for the explicit test and the call in optimize.minimize(). The reason is that scipy.optimize.minimize() expects the initial guess initial_theta as a keyword argument and therefore it has to be given before the other arguments x,Y are given. Since you want to optimize for theta, I suggest that you change the order of the arguments in costFunction() and, accordingly, the call of optimize.minimize(). Here is a working example:

from scipy import optimize
import numpy as np

def sigmoid(t):
    return 1./(1. + np.exp(t))

X = np.random.random(size=(1000,3))
Y = np.random.random(size=(1000))

def costFunction(theta, x,y):
    J = 0.0
    m = y.size
    J = -1/m * np.sum(((1-y)*np.log(1-sigmoid(np.dot(x,theta))))+((y)*np.log(sigmoid(np.dot(x,theta)))))
    grad = 1/m*np.dot(x.T, (sigmoid(np.dot(x, theta))-y))
    return J, grad

#To check the function :
print(X[:,:3].shape)
J,grad = costFunction(theta=np.asarray([0,0,0]), x=X[:,:3],y=Y)
print(J)
print( grad)

options = {'maxiter' : 400}
initial_theta = np.zeros(3)
x = X[:,:3]
res = optimize.minimize(costFunction,
                        x0 = initial_theta,
                        args=(x, Y),
                        jac=True,
                        method='TNC',
                        options=options)

cost = res.fun
thetaresult = res.x
print(cost)
print(thetaresult)

How to fix Value error reported when using optimize.minimize() but the function referenced has no error?

1 Answers