Backpropogation neural network - error not converging

Question

I am using backpropogation algorithm for my model. It works perfectly fine a simple xor case and when I tested it for a smaller subset of my actual data.

There are 3 inputs in total and a single output(0,1,2)
I have split the data set into training set (80% amounting to approx 5.5k) and the rest 20% as validation data.

I use trainingRate and momentum for calculating the delta weights.

I have normalized the input as below

from sklearn import preprocessing

min_max_scaler = preprocessing.MinMaxScaler()
X_train_minmax = min_max_scaler.fit_transform(input_array)

I use 1 hidden layer with sigmoid and linear activation functions for input-hidden and hidden-output respectively.

I train with trainingRate = 0.0005, momentum = 0.6, Epochs = 100,000. Any higher trainingRate shoots up the error to Nan. momentum values between 0.5 and 0.9 works fine and any other value makes the error Nan.

I tried various number of nodes in the hidden layer such as 3,6,9,10 and the error converged to 4140.327574 in each case. I am not sure how to reduce this. Changing the activation functions doesn't help. I even tried adding another hidden layer with gaussian activation function but I cannot reduce the error whatsoever.

Is it because of the outliers? Do i need to clean those values from the training data?

Any suggestion would be of great help be it the activation function, hidden layers, etc. I had been trying to get this working for quite some time and I am sort of stuck now.

to test your hypothesis, try adding outliers to the simple cases and see how it reacts — SlimJim
The error did reduce to ~ 1.5k on some cleanup of outliers. However, the model isn't good yet because of the data, my applcation of the backpropogation algo or maybe both. — user2489122

gkapellmann gkapellmann · Accepted Answer · 2014-03-13T13:29:29

Well I'm having kind of a similar problem, still haven fixed it, but I can tell you a couple of things I have found. I think the net is overfitting, my error at some point goes down and then starts going up again, also the verification set... is this you case also?

Check if you are implementing well the "early stopping" algorithm, most of the times the problem is not the backpropagation, but the error analysis or the validation analysis.

Hope this helps!

Backpropogation neural network - error not converging

1 Answers