Python simple neural network code not working

Question

I'm trying to learn about neural networks and coded a simple back-propagation neural network that uses sigmoid activation functions and random weight initialisation. I was trying multiplication with two input values 3 and 2 in the input layer and target output 6 in output layer. When I execute my code the value for w1 and w2 keeps on increasing and doesn't stop at the correct value.

I am new to both Python and neural networks and I'd appreciate assistance.

import numpy as np
al0 = 3
bl0 = 2 
import random
w1 =random.random()
w2 =random.random()
b = 0.234
ol1 = 6
def sigm(x,deriv=False):
   if deriv==True:
       return x*(1-x)
   return 1/(1+np.exp(-x))
y = sigm(x)
E = 1/2*(ol1 - y)**2
dsig = sigm(x,True)
dyE = y-ol1

for iter in range(10000):
    syn0 = al0*w1
    syn1 = bl0*w2
    x = syn0 + syn1 + b
    dtotal1 = dyE*dsig*al0
    w1 = w1 + 0.01*dtotal1
    dtotal2 = dyE*dsig*bl0
    w2 = w2 + 0.01*dtotal2
w1
w2

It would appear as though you are using x before it is assigned. — Matt Cremeens
And I would expect w1 and w2 to continue to increase as it would appear you continually add a positive amount to them in your loop. — Matt Cremeens

bjschoenfeld bjschoenfeld · Accepted Answer · 2018-08-16T18:28:56

First you need to get your code in order. These lines

y = sigm(x)
E = 1/2*(ol1 - y)**2
dsig = sigm(x,True)
dyE = y-ol1

need to happen inside the for loop, after x = syn0 + syn1 + b.

Next, there are a few neural network items to address. Here is a technical description of backprop.

The derivative of sigm(x) is sigm(x) * (1 - sigm(x)), or in your case, y * (1 - y), which as you have implemented is sigm(y,True).

Your bias also needs to be updated as well. This is a critical part of the network and is a learned parameter. You could use:

dtotalb = dyE*dsig*1
b = b - 0.01*dtotalb

The multiply by 1 is unnecessary, but instructive. This is the derivative of the "net" term, which you called x, with respect to b, i.e. 1.

You may have noticed that the update I gave for b uses a - instead of +. The very last line of this section in the link above shows this is necessary to make sure the updates go in the correct direction to minimize your error.

Finally, consider the possible values your network can output. The final output y is the result of calling sigm(x). The values that y can take on are in the open interval (0,1). Your network however, is trying to learn the value of 6. The closest it can get is almost 1. So as you continue to iterate, the weights will continue to increase to try and increase the output value of the sigmoid. The weights will grow indefinitely.

With all these changes, try making your target something in (0,1), e.g. 0.6. I am able to get the E to decrease to near 0 and your weights to converge when I make all these changes.

Side note: In order for your network to learn a value of 6, you would need another layer of weights without an activation function. Alternatively, you could remove the activation function, but then you start to lose the neural network approach you are trying to learn.

Python simple neural network code not working

1 Answers