My two layer neural network model doesn't converge

Question

I am training a two layer neural network. I waited for 15000 epochs, still model doesn't converge.

ans = []
for i in range(1000):
    x1,y1 = random.uniform(-3,3),random.uniform(-3,3)
    if x1*x1 + y1 * y1 < 1:
        ans.append([x1,y1,0])
    elif x1*x1 + y1 * y1 >= 2 and x1*x1 + y1 * y1 <=8:
        ans.append([x1,y1,1])

data = pd.DataFrame(ans)
print(data.shape)
X = np.array(data[[0,1]])
y = np.array(data[2])

I am generating random points generating data. the data looks like something like this.

weights_layer1 = np.random.normal(scale=1 / 10**.5, size=(2,20))
bias1 = np.zeros((1,20))
bias2 = np.zeros((1,1))
weights_layer2 = np.random.normal(scale=1 / 10**.5, size=(20,1))
for e in range(15000):
    for x,y1 in zip(X,y):
         x = x.reshape(1,2)
         layer1 = sigmoid(np.dot(x,weights_layer1)+bias1)
         layer2 = sigmoid(np.dot(layer1,weights_layer2)+bias2)
        
        dk = (y1-layer2)*layer2*(1-layer2)
        dw2 = learnrate * dk * layer1.T
        dw2 =dw2.reshape(weights_layer2.shape)
   # print(dw2.shape)
    
    
        weights_layer2 += dw2
   # bias2 += dk * learnrate
    
        dj = weights_layer2.T* layer1*(1-layer1)*dk
        dw1 = learnrate * np.dot(x.T,dj)

I am calculating loss in this manner.

loss = 0
for x,y1 in zip(X,y):
    layer1 = sigmoid(np.dot(x,weights_layer1))
    layer2 = sigmoid(np.dot(layer1,weights_layer2))
    loss += (layer2 - y1)**2
    
    
print(loss)

cant find what is going wrong,can you see anything? Thanks. I trained the same with pytorch it is converging fine.

the final model looks like this on trained data. but on test data it is worse.

There is no way anyone can help with this unless you also share the data and the full code. — Ananda
x is just set of points which are inside and outside a circle, it has to detect if the point is inside or outside — Anil M
I am generating random inputs and checking accuracy on random points. — Anil M
Please take a look at the guidelines for asking a question. If you expect others to do work just to be able to reproduce your issue, they are simply going to move on to the next question. You need to help us help you. stackoverflow.com/help/how-to-ask. When asking a quesiton, please post the full question, including any code you have for generating data etc. — Ananda

Anil M Anil M · Accepted Answer · 2021-01-28T16:39:08

After few hours of trying out, I found the problem. This network doesn't converge without biases. Used biases it converged in 5000 epochs.

My two layer neural network model doesn't converge

1 Answers