Logistic Regression Implementation

Question

I am having some difficulties in implementing logistic regression, in terms of how should I should proceed stepwise. According to what I have done so far I am implementing it in the following way:

First taking theta equal to the number of features and making it a n*1 vector of zeros. Now using this theta to compute the the following htheta = sigmoid(theta' * X');
theta = theta - (alpha/m) * sum (htheta' - y)'*X
Now using the theta computed in the first step to compute the cost function
J= 1/m *((sum(-y*log(htheta))) - (sum((1-y) * log(1 - htheta)))) + lambda/(2*m) * sum(theta).^2
In the end computing the gradient
grad = (1/m) * sum ((sigmoid(X*theta) - y')*X);

As i am taking theta to be zero. I am getting same value of J throughout the vector, is this the right output?

Could you provide a minimum working example, for instance using some dummy data. You can produce it like this: y=randi(2,[10,1])-1 & X=[ones(10,1) randn([10,1]). From what I can see: the cost function should change as you update theta. However, it is unclear why you compute the gradient twice (i.e., once at updating theta and in the final step). — jvz
@zundertj actually that is the part i am also unclear about. If you can tell me which thing to compute first and why, that will do. — user3593450

jvz jvz · Accepted Answer · 2017-08-16T07:47:38

You are computing the gradient in the last step, while it has been computed before in the computation of the new theta. Moreover, your definition of the cost function contains a regularization parameter, but this is not incorporated in the gradient computation. A working version without the regularization:

% generate dummy data for testing
y=randi(2,[10,1])-1;
X=[ones(10,1) randn([10,1])];

% initialize
alpha = 0.1;
theta = zeros(1,size(X,2));
J = NaN(100,1);

% loop a fixed number of times => can improve this by stopping when the
% cost function no longer decreases
htheta = sigmoid(X*theta');
for n=1:100
    grad = X' * (htheta-y); % gradient
    theta = theta - alpha*grad'; % update theta
    htheta = sigmoid(X*theta');
    J(n) = sum(-y'*log(htheta)) - sum((1-y)' * log(1 - htheta)); % cost function
end

If you now plot the cost function, you will see (except for randomness) that it converges after about 15 iterations.

Logistic Regression Implementation

1 Answers