Linear regression, gradient descent not working for some values

Question

I have implemented linear regression with gradient descent but it doesn't work when I need to get decreasing function or constant function.

It works for data like

x_train = np.array([30,60,70,100])
y_train= np.array([60,120,145,195])

where I need to get increasing function like here f(x_train) ≈ 2*x_train=y_train or for data like

x_train = np.array([30,60,70,100])
y_train= np.array([30,60,70,100])

where is f(x_train)=x_train=y_train, but it doesn't work for data like this

x_train = np.array([50,100,150,200])
y_train= np.array([150,100,50,0])

decreasing function f(x_train)=200-x_train=y_train or for constant function f(x_train)=100=y_train

x_train = np.array([50,100,150,200])
y_train= np.array([100,100,100,100])

import numpy as np
import matplotlib.pyplot as plt

#doesnt work
#x_train = np.array([50,100,150,200])
#y_train= np.array([150,100,50,0])

#work
x_train = np.array([30,60,70,100])
y_train= np.array([60,120,145,195])

def model(x,w,b):
    return x*w+b;

def cost(y,y_hat):
    return np.sum((y-y_hat)**2)

learning_rate=0.000001
def trainning_round(x_train,y_train,w,b,learning_rate):

    y_hat=model(x_train,w,b)
    print(cost(y_train,y_hat))

    w_gradient=-2*x_train.dot(y_train-y_hat)
    b_gradient=-2*np.sum(y_train-y_hat)

    w-=learning_rate*w_gradient
    b-=learning_rate*b_gradient

    return w,b

num_epoch=200
def train(X,Y):

    w=0
    b=0

    #for plt
    ar = np.arange(0, 200, 0.5)
    def f(t):
        return t*w+b

    for i in range(num_epoch):
            w,b=trainning_round(X,Y,w,b,learning_rate)
            plt.plot(ar,f(ar))
            plt.axis([0, 200, 0, 200])

    plt.plot(X, Y, 'ro')

train(x_train,y_train)
plt.show()

I have tried some other algorithms but they didn't work. Thanks in advance

Welcome to StackOverflow! Please have a look at How do I ask a good question? and how to construct a minimal reproducible example, then edit your post accordingly. Currently, your question is not very clear, and also you haven't provided the data needed to reproduce your problem. In order to avoid having your question closed, please update to be clearer and more specific. — andrew_reece

Junhee Shin Junhee Shin · Accepted Answer · 2018-07-18T03:00:52

I modified some parts. It is important that you decide appropriate learning_rate and iteration epoch. That's some difficult. So we use a framework like tensorflow.

import numpy as np
import matplotlib.pyplot as plt

#doesnt work
x_train = np.array([50,100,150,200])
y_train= np.array([150,100,50,0])

#work
# x_train = np.array([30,60,70,100])
# y_train= np.array([60,120,145,195])

def model(x,w,b):
    return x*w+b;

def cost(y,y_hat):
    return np.sum((y-y_hat)**2)/y.size

learning_rate=0.0001
def trainning_round(x_train,y_train,w,b,learning_rate):

    y_hat=model(x_train,w,b)
    j = cost(y_train,y_hat)

    # w_gradient=-2*x_train.dot(y_train-y_hat)
    # b_gradient=-2*np.sum(y_train-y_hat)
    w_gradient=x_train.dot(y_hat-y_train) / y_train.size
    b_gradient=np.sum(y_hat-y_train) / y_train.size

    print(w_gradient, b_gradient)

    w=w-learning_rate*w_gradient
    b=b-learning_rate*b_gradient
    print(j, w,b)
    return w,b

num_epoch=200000
def train(X,Y):

    w=2.1
    b=1.5

    #for plt
    ar = np.arange(0, 200, 0.5)

    for i in range(num_epoch):
        w,b=trainning_round(X,Y,w,b,learning_rate)

    plt.plot(ar,model(ar, w, b))
    plt.axis([0, 300, 0, 200])

    plt.plot(X, Y, 'ro')

train(x_train,y_train)
plt.show()

Linear regression, gradient descent not working for some values

2 Answers