Neural networks: why does it work worse when you give it more neurons?

0
votes

I just coded my first neural network and experimented a little with it... It's task is very simple: it should basically output the rounded number. It consists of one input neuron and one output neuron with 1 hidden layer consisting of 2 hidden neurons. At first I gave it about 2000 random generated training data sets.

When I gave it 3 hidden layers consisting of 10 hidden neurons. The results started to get worse and even after 10000 training sets it still output many wrong answers. The neural network with 2 hidden neurons worked way better.

Why does this happen? I thought the more neurons a neural network had, the better it would be...

So how do you find the best number of neurons and hiddenlayers?

1
neural-network
Can you share a little of your code, and specifically the training data you are experimenting with? Generally, the provided answer is correct in that you should see a drop in performance due to overfitting, but your network (even with three layers) seems way to small to do so, unless you are looking at a trivial task. – dennlinger
My problem is not that the performance drops... even after 10000 training sets the amount of wrong answers is a lot higher when i use more neurons. – IceRevenge

1 Answers

4
votes

If by "worse" you mean less accuracy on a test set, the problem is most likely overfitting.

In general, I can tell you this: More layers will fit more complicated functions on data. Maybe your data resembles a lot a straight line, so a simple linear function will do great. But imagine you try fitting a 6-th degree polynomial to the data. As you may know, high degree even functions go to infinity (+-) very fast, so this high degree model will predict too large values at the extremes.

In summary, your problem is most likely overfitting (high variance). You can check out several more intuitive explanations on the bias-variance tradeoff somewhere with graphs.

quick google search: https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff