from Octave's fminunc to Julia's optimize with gradient function with parameters

Question

I am passing my code from octave to julia, in this case a logistic regression. The gradient function takes in addition to the initial theta, X with my features and Y with the sought values.

in Octave works

function [J, grad] = costFunction(theta, X, y)
options = optimset('GradObj', 'on', 'MaxIter', 400);
[theta, cost] = fminunc(@(t)(costFunction(t, X, y)), initial_theta, options);

on Julia first try

optimize(t->CostFunction(t, X1, y), initial_theta, BFGS())

MethodError: no method matching -(::Tuple{Float64,Array{Float64,2}}, ::Tuple{Float64,Array{Float64,2}})

so I separated the function in two: Cost and Gradient

function CostFunction2(theta, X, y)
J = 0;
#m = length(y);
m = size(y,1);
grad = zeros(size(theta));
J = 1/m * sum( (-y .* log.(sigmoid(X*theta))) - ((1.0 .- y) .* log.(1.0 .- sigmoid(X*theta))) );
return J;
end


function Gradient2(X, y, theta)
grad = zeros(size(theta));
grad = (1/m) .* (sum((sigmoid(X*theta).-y) .* X, dims=1))';
return grad;
end

I put only the cost function and it worked, but I don't have the last values of theta. I don't know how to get it

optimize(t->CostFunction2(t, X1, y), initial_theta, BFGS())

I tried this but it didn't work, and I can't find any reference that says how or brings an example

optimize(t->CostFunction2(t, X1, y), Gradient2(X, y, t), initial_theta, BFGS())

UndefVarError: t not defined

How can I obtain the obtained theta values? and How can I include my own gradient function with various parameters?

I hope you can help me, thank you very much

can you please post your whole code in once piece. It seems that there is some simple problem with it (most likely that you are mixing scalars and vectors in an incorrect way). In particular note that in the call to optimize the initial_theta variable should be a vector not a scalar. — Bogumił Kamiński
data = load('data.txt'); X = data(:, [1, 2]); y = data(:, 3); [m, n] = size(X); X = [ones(m, 1) X]; initial_theta = zeros(n + 1, 1); function [J, grad] = costFunction(theta, X, y) J = 0; m = size(y,1); grad = zeros(size(theta)); J = -sum( (y .* log(sigmoid(X*theta))) + ((1-y) .* log(1 - sigmoid(X*theta))) ) / m; grad = (1/m) * (sum((sigmoid(X*theta)-y) .* X))'; end options = optimset('GradObj', 'on', 'MaxIter', 400); [theta, cost] = fminunc(@(t)(costFunction(t, X, y)), initial_theta, options); fprintf('Cost:%f theta:%f \n',cost,theta); — Nemachtiani
` using DelimitedFiles; Data = readdlm("ex2data1.txt", ',', Float64); X = Data[:,1:2]; y = Data[:,3]; m = length(y); `` m,n = size(X); X = [ones(m,1) X]; initial_theta = zeros(n + 1, 1); `` function CostFunction2(theta, X, y) J = 0; #m = length(y); m = size(y,1); grad = zeros(size(theta)); J = 1/m * sum( (-y .* log.(sigmoid(Xtheta))) - ((1.0 .- y) . log.(1.0 .- sigmoid(Xtheta))) ); return J; end function Gradient2(X, y, theta) grad = zeros(size(theta)); grad = (1/m) . (sum((sigmoid(Xtheta).-y) . X, dims=1))'; return grad; end ` — Nemachtiani
` using Optim; optimize(t->CostFunction2(t, X1, y), Gradient2(X, y, t), initial_theta, BFGS()); ` — Nemachtiani
Sorry for the code format. The first part is how I did it in Octave, the other two is how I did it in Julia. Thank you — Nemachtiani

Bogumił Kamiński Bogumił Kamiński · Accepted Answer · 2020-10-25T07:25:09

I have cleaned typos in your code, added missing definitions, and removed unnecessary operations.

After these changes the following works:

using DelimitedFiles

Data = readdlm("ex2data1.txt", ',', Float64)
X = Data[:,1:2]
y = Data[:,3]
m, n = size(X)
X = [ones(m) X]
initial_theta = zeros(n + 1)

sigmoid(x) = 1 / (1 + exp(-x))

function CostFunction2(theta, X, y)
    Xt = X*theta # do this step outside as it is repeated twice otherwise
    return sum(-y .* log.(sigmoid.(Xt)) .- (1.0 .- y) .* log.(1.0 .- sigmoid.(Xt)))
end

using Optim
optimize(t->CostFunction2(t, X, Y), initial_theta, BFGS())

from Octave's fminunc to Julia's optimize with gradient function with parameters

1 Answers