Issues with neural network

Question

I am having some issues with using neural network. I am using a non linear activation function for the hidden layer and a linear function for the output layer. Adding more neurons in the hidden layer should have increased the capability of the NN and made it fit to the training data more/have less error on training data.

However, I am seeing a different phenomena. Adding more neurons is decreasing the accuracy of the neural network even on the training set. enter image description here

Here is the graph of the mean absolute error with increasing number of neurons. The accuracy on the training data is decreasing. What could be the cause of this?

Is it that the nntool that I am using of matlab splits the data randomly into training,test and validation set for checking generalization instead of using cross validation.

Also I could see lots of -ve output values adding neurons while my targets are supposed to be positives. Could it be another issues?

I am not able to explain the behavior of NN here. Any suggestions? Here is the link to my data consisting of the covariates and targets

https://www.dropbox.com/s/0wcj2y6x6jd2vzm/data.mat

First of all, is the error in range of 10^7 usual? Isn't that too high? This may indicate that your neural network is not training at all. Have you seen the training evolution from your neural network to see if it is really training? — Werner
@Werner. The error scale is fine, it's supposed to be in that range because my outputs are very high values. So it's normal to see the error in that range — user34790
Are you using the default mapminmax normalization? Matlab applies the normalization by default, and (I might be wrong) the error scale is also normalized. If you have very high outputs and do not normalize it will saturate your non linear activation functions (if you use any) which would explain this non expected behavior (adding more saturated neurons would saturate even more the output). Dont forget to check your error evolution during the training stage to see if your networks are really training. — Werner
@Werner. Yes, I am using the default mapminmax normalization. Yeah, about the error, I couldn't see the error dropping much in most of the cases. I am learning like 100 nn for different areas and averaging the error. In some stations, I could see the error decreasing. However, it's very small, I mean the error change — user34790
So, I would say that you forget the cross validation for now. First you need one neural network with a good training shape, start with 5 hidden neurons and see if you can get one working (don't forget to do this always before you start the cross validation step, the cross validation is less important than this step itself). What kind of activation on your neurons are you using? Are you using anything else than the default matlab neural network? Be sure to pay attention to your dataset, there is probably something wrong there. — Werner

idfah idfah · Accepted Answer · 2013-09-04T18:41:43

I am unfamiliar with nntool but I would suspect that your problem is related to the selection of your initial weights. Poor initial weight selection can lead to very slow convergence or failure to converge at all.

For instance, notice that as the number of neurons in the hidden layer increases, the number of inputs to each neuron in the visible layer also increases (one for each hidden unit). Say you are using a logit in your hidden layer (always positive) and pick your initial weights from the random uniform distribution between a fixed interval. Then as the number of hidden units increases, the inputs to each neuron in the visible layer will also increase because there are more incoming connections. With a very large number of hidden units, your initial solution may become very large and result in poor convergence.

Of course, how this all behaves depends on your activation functions and the distributio of the data and how it is normalized. I would recommend looking at Efficient Backprop by Yann LeCun for some excellent advice on normalizing your data and selecting initial weights and activation functions.

Issues with neural network

1 Answers