Cost function of logistic regression outputs NaN for some values of theta

Question

While implement logistic regression with only numpy library, I wrote the following code for cost function:

#sigmoid function
def sigmoid(z):
  sigma = 1/(1+np.exp(-z))
  return sigma
#cost function
def cost(X,y,theta):
  m = y.shape[0]
  z = X@theta
  h = sigmoid(z)
  J = np.sum((y*np.log(h))+((1-y)*np.log(1-h)))
  J = -J/m
  return J

Theta is a (3,1) array and X is the training data of shape (m,3). First column of X is ones. For theta = [0,0,0], cost function outputs 0.693 which is the correct cost, but for theta = [1,-1,1], it outputs:

/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: RuntimeWarning: divide by zero encountered in log
  """
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: RuntimeWarning: invalid value encountered in multiply
  """
nan

My code for gradient descent is:

#gradientdesc function
#alpha is the learning rate, iter is the number of iterations
def gradientDesc(X,y,theta,alpha,iter):
  m = y.shape[0]
  #d represents the derivative term
  d = np.zeros((3,1))
  for iter in range(iter):

    h = sigmoid(X@theta) - y
    temp = h.T.dot(X)

    d = temp.T
    d/=m
    theta = theta - alpha*d

  return theta

But this does not give the correct value of theta. What should I do?

Varun Nayak Varun Nayak · Accepted Answer · 2020-04-25T07:23:49

Are the values in X large? This might lead to the sigmoid returning values close to zero that lead to the warnings you are seeing. Have a look at this thread: Divide-by-zero-in-log

Your gradient descent won't work properly unless you solve this issue of values exploding. I would also consider adding regularization in your cost function.

J += C * np.sum(theta**2)

Cost function of logistic regression outputs NaN for some values of theta

1 Answers