Full Batch, Stochastic and Mini Batch gradient descent in Python, Linear Regression

Question

I'm trying to understand and implement these algorithms in Python. I used for that purpose sklearn.linear_model.SGDRegressor and my code looks like this:

import numpy as np
from sklearn import linear_model
from sklearn.metrics import mean_squared_error
from math import sqrt

X = np.array([1,2,4,3,5]).reshape(-1,1)
y = np.array([1,3,3,2,5]).reshape(-1,1).ravel()

Model = linear_model.SGDRegressor(learning_rate = 'constant', alpha = 0, eta0 = 0.01, shuffle=True, max_iter = 4)

Model.fit(X,y)
y_predicted = Model.predict(X)

mse = mean_squared_error(y, y_predicted)
print("RMSE: ", sqrt(mse))
print("The intercept is:", Model.intercept_)
print("The slope is: ", Model.coef_)

and I got the following results:

RMSE:  0.7201328561288026
The intercept is: [ 0.21990009]
The slope is:  [ 0.79460054]

Based on this article: https://machinelearningmastery.com/linear-regression-tutorial-using-gradient-descent-for-machine-learning/ the results are quite similar, so I guess everything's ok.

Now I've tried to implement the following code:

from sklearn import linear_model
import numpy as np
from sklearn.metrics import mean_squared_error
from math import sqrt

X = np.array([1,2,4,3,5]).reshape(-1,1)
y = np.array([1,3,3,2,5]).reshape(-1,1).ravel()

numtraining = len(X)

def iter_minibatches(chunksize):
    # Provide chunks one by one
    chunkstartmaker = 0
    while chunkstartmaker < numtraining:
        chunkrows = range(chunkstartmaker, chunkstartmaker+chunksize)
        X_chunk = X[chunkrows] 
        y_chunk = y[chunkrows]
        yield X_chunk, y_chunk
        chunkstartmaker += chunksize

batcherator = iter_minibatches(chunksize=1)

Model = linear_model.SGDRegressor(learning_rate = 'constant', alpha = 0, eta0 = 0.01, shuffle=True, max_iter = 4)

for X_chunk, y_chunk in batcherator:

    Model.partial_fit(X_chunk, y_chunk, np.unique(y_chunk))

y_predicted = Model.predict(X)

mse = mean_squared_error(y, y_predicted)

print("RMSE: ", sqrt(mse))
print(Model.coef_)
print(Model.intercept_)

and I got the following results:

RMSE:  1.1051202460564218
[ 1.08765043]
[ 0.29586701]

As far as I understand from the theory: When chunksize = 1, mini batch gradient descent is the same as stochastic gradient descenet. This is not the case in my code.. Whether the code is wrong or I miss something?

Aleksandar Djuric Aleksandar Djuric · Accepted Answer · 2018-02-23T06:51:03

I'm not entirely sure whats going on but converting batcherator to a list helps.

Also, to properly implement minibatch gradient descent with SGDRegressor, you should manually iterate through your training set (instead of setting max_iter=4). Otherwise SGDRegressor will just do gradient descent four times in a row on the same training batch. Furthermore, you can shuffle the training batches for even more randomness.

...

Model = linear_model.SGDRegressor(learning_rate = 'constant', alpha = 0, eta0 = 0.01, shuffle=True)

chunks = list(batcherator)
for _ in range(4):
    random.shuffle(chunks)
    for X_chunk, y_chunk in chunks:
        Model.partial_fit(X_chunk, y_chunk)

y_predicted = Model.predict(X)

...

This yields:

RMSE: 0.722033757406
The intercept is: 0.21990252
The slope is: 0.79236007

Full Batch, Stochastic and Mini Batch gradient descent in Python, Linear Regression

1 Answers