What is wrong with this gradient descent algorithm?

Question

X_train is already normalized using StandardScaler() and also the categorical columns have been converted into one hot encodings.

X_train.shape=(32000, 37)

I am using the following code to compute the value of w using gradient descent

w = np.zeros(len(X_train.columns))
learning_rate = 0.001    
for t in range(1000):
    Yhat = X.dot(w)
    delta = Yhat - Y_train
    w = w - learning_rate*X_train.T.dot(delta)

My w vector blows up (i.e. increases very fast) and and every entry of w becomes NaN. I tried reducing the number of epochs to 10, 15, 20, etc., and what I found is every element of w is diverging instead of converging.

I tried using normal equations and the w is indeed coming up fine in that case (line breaks added for readability):

w_found_using_normal_eqns = [ 3.53175449e-14  1.27924991e-14 -5.42441539e-14
9.91098366e-16 -2.31752259e-14 -6.21205773e-13  1.66139358e-13
2.72739782e-13 -1.65076881e-13 -1.25280166e-14 -1.98905983e-14  3.78837632e-13
-1.39424696e-12 -6.48511452e-15  1.58136412e-14  1.39778439e-12
-1.06142667e-14  3.00624557e-14 -1.70159700e-15 -6.91500349e-15 -4.04842208e-15
2.37516654e-16  3.25211677e+01 -2.86074823e+01 -2.86074823e+01
-2.86074823e+01 -2.86074823e+01 -2.86074823e+01 -2.86074823e+01 -2.86074823e+01 
3.55024823e+01  3.55024823e+01 3.55024823e+01  3.55024823e+01  
3.55024823e+01  3.55024823e+01 3.55024823e+01]

The r^2 error is 1 if I use normal equations to solve for w.

daniLab daniLab · Accepted Answer · 2018-08-06T12:39:08

The gradient descent weights update formula is normalised by training set size.

In the last row you need to divid the learning rate by the training set size.

the fix code:

w = w - (learning_rate/X_train.shape) * X_train.T.dot(delta)

What is wrong with this gradient descent algorithm?

1 Answers