Logistic regression does not work in a simple example

Question

I think I understand logistic regression (from the book of Bishop on page 206), but when I do the following simple example, I cannot find the (absolute) minimum of the cost function (error function).

My trainingset consists of 2 points on the real number line: 1 and -1 which have the (classification) values 1 respectively 0.

The cost function is, where sis the sigmoid function, and where the sum is taken over the whole trainingset:

E(w0,w1)= sum -t.log(s(w0+w1.x)-(1-t).log(1-s(w0+w1.x))

So here in our case:

E(w0,w1)= -log(s(w0+w1))-log(1-s(w0-w1))

I cannot find the mimimum. If I use an algorithm, w1 goes to infinity.

I thought that w0=0. If you suppose that, then because 1-s(-w)=s(w), we get

E(w1)=-2.log(s(w))

which is a descending function to zero, so the minimum is at infinity.

Answer 1

I think the problem is that your function has no global minimum wrt w0,w1. It could be proved with equations system:

dE / dw0 = exp(w1)(exp(2w0) - 1) / ((exp(w0) + exp(w1)) * (exp(w0 + w1) + 1))
dE / dw1 = (-exp(w1 + 2w0) - exp(w1) - 2exp(w0)) / ((exp(w0) + exp(w1)) * (exp(w0 + w1) + 1))

dE / dw0 = 0 when w0 = 0. In this case dE / dw1 = 0 is not feasible.

However if you add regularization term to your loss function, i.e. Enew = E + w1^2, you'll find optima.