Log Likelihood function for Machine Learning Algorithm

0
votes

I am implementing logistic regression in Python and am at the point where I am writing code for maximum likelihood estimation for a parameter vector according to this formula:

enter image description here

I have implemented it in the following manner:

-np.sum(np.log(1 + np.exp(-1*(y*x.dot(w.T)))))

In a function where w is the weight vector, x is the feature matrix, and y is the label vector. I am not obtaining the classification I expect based on test data and suspect it has something to do with the way I've implemented this log-likelihood equation. I run this function in scipy.minimize.optimize where the additional paramters are the initial weight estimates (all initialized to zero) and args = (x,y). In terminal, I get the following errors:

RuntimeWarning: overflow encountered in exp; return -np.sum(np.log(1 + np.exp(-1*(y*x.dot(w.T)))))

RuntimeWarning: invalid value encountered in subtract df = fun(x) - f0

It is a simple binary classification task. Can someone provide some hints as to why this may be happening and how I can address it in the most efficient manner possible?

EDIT: A colleague of mine suggested that this same operation could be done using logsumexp() in that it is more numerically stable, but I'm not sure how this is possible so the equation requires a sum of logs but logsumexp() is a log of sums. Any suggestions?

1
pythonmachine-learninglogistic-regressionlog-likelihood
You need to post an example of the inputs that are causing the same issue. You can't pose the entire data likely, so you need to identify a small sub-sample of the data that causes the same issue. See proving a MRE here. stackoverflow.com/help/how-to-ask - Ananda

1 Answers

0
votes

The reason is that when -1*(y*x.dot(w.T)))) is too large, funtion exp will overflow. I test on my x64 machine and find when the parameter for exp is larger than 750, exp will overflow:

>>> np.exp(750)
inf
>>>

Thus I suggest a normalization applyed on weight w and input x.