I am currently building a nn for a dataset with 387 features and 3000 samples. The outputs are 3 classes. I configured the network structure as following:
input->200->{300->100}->50->output
Did I choose the correct number of nodes and layers? How to determine the number of nodes of each layers(input,hidden and output)? Is there any rule?
Rules? As many as you want and none. Here is an excerpt from the Neural Network FAQ which is a good page to consult for basic questions:
A: How many hidden units should I use? ==========================================
There is no way to determine a good network topology just from the number of inputs and outputs. It depends critically on the number of training examples and the complexity of the classification you are trying to learn. There are problems with one input and one output that require millions of hidden units, and problems with a million inputs and a million outputs that require only one hidden unit, or none at all.
Some books and articles offer "rules of thumb" for choosing a topopology -- Ninputs plus Noutputs dividied by two, maybe with a square root in there somewhere -- but such rules are total garbage. Other rules relate to the number of examples available: Use at most so many hidden units that the number of weights in the network times 10 is smaller than the number of examples. Such rules are only concerned with overfitting and are unreliable as well.
In your case, however, one can definitely say that the network is much too complex (even if you applied strong regularization). Why so many hidden layers? Start with one hidden layer -- despite the deep learning euphoria -- and with a minimum of hidden nodes. Increase the hidden nodes number until you get a good performance. Only if not I would add further layers. Further, use cross validation and appropriate regularization.
As Yoshua Bengio, Head of Montreal Institute for Learning Algorithms remarks:
"Very simple. Just keep adding layers until the test error does not improve anymore."
A method recommended by Geoff Hinton is to add layers until you start to overfit your training set. Then you add dropout or another regularization method.
For your task:
As they said, there is no "magic" rule to calculate the number of hidden layers and nodes of Neural Network, but there are some tips or recomendations that can helps you to find the best ones.
The number of hidden nodes is based on a relationship between:
To minimize the error and have a trained network that generalizes well, you need to pick an optimal number of hidden layers, as well as nodes in each hidden layer.
Too few nodes will lead to high error for your system as the predictive factors might be too complex for a small number of nodes to capture
Too many nodes will overfit to your training data and not generalize well
You could find some general advices on this page:
Section - How many hidden units should I use?
If your data is linearly separable then you don't need any hidden layers at all. Otherwise there is a consensus on the performance difference from adding additional hidden layers: the situations in which performance improves with a second (or third, etc.) hidden layer are very small. Therefore, one hidden layer is sufficient for the large majority of problems.
There are some empirically-derived rules-of-thumb, of these, the most commonly relied on is 'the optimal size of the hidden layer is usually between the size of the input and size of the output layers'.
In sum, for most problems, one could probably get decent performance by setting the hidden layer configuration using just two rules:
