I'm researching on the topic of "logistic Regression" in machine learning.I could understand the entire concept that it's trying to maximize the likelihood of an instance belonging to a particular class label
The algorithm, if run for many iterations, finds a weight vector that separates the instances and then keeps increasing the magnitude of the weight vector. I donot understand why would it try to increase the magnitude of weight vector
Any Help would be highly appreciable!
I'm guessing your data is linearly separable? IIRC, logistic regression breaks down in that case. I think this is a well-known problem. Quoting from here (the first hit on google for "logistic regression separable data"):
...However, when the training data is linearly separable, two bad things happen: 1. |θ| goes to infinity; 2. There are infinite number of MLE’s. To see this, note any step function (sigmoid with |θ|=∞) that is in the gap between the two classes is an MLE
One way to avoid this is to incorporate a prior on θ in the form of a zero-mean Gaussian with covariance 1/(2λ)I
I think what you are asking about is Regularization in Machine learning, From my understanding of it this is done so as to avoid the phenomena known as overfitting i.e when the hypothesis fits the training data almost perfectly at the cost of giving a poor hypothesis for the test data.
I hope this helps to an extent.
