Neural Network Outputs Are Not Changing Very Much

Question

I have 20 output neurons on a feed-forward neural network, for which I have already tried varying the number of hidden layers and number of neurons per hidden layer. When testing, I've noticed that while the outputs are not always exactly the same, they vary from test case to case very little, especially in respect to one another. It seems to be outputting nearly (within 0.0005 depending on the initial weights) the same output on every test case; the one that is the highest is always the highest. Is there a reason for this?

Note: I'm using a feed-forward neural network, with resilient and common backpropagation, separating training/validation/testing and shuffling in between training sets.

UPDATE: I'm using the network to categorize patterns from 4 inputs into one of twenty output possibilities. I have 5000 training sets, 800 validation sets, and 1500 testing sets. Number of rounds can vary depending on what I'm doing, on my current training case, the training error seems to converge too quickly (under 20 epochs). However, I have noticed this non-variance at other times when the error will decrease over a period of 1000 epochs. I have also adjusted the learning rate and momentum for the regular propagation. Resilient propagation does not use a learning rate or momentum for updates. This is being implemented using Encog.

Answer 1

Your dataset seems problematic to begin with. 20 outputs for 4 inputs seem too many. The number of output is generally much smaller than the number of inputs. Most probably, either the dataset is wrongly formulated, or you have misunderstood something in the problem you are trying to solve. Anyway, some things regarding your other comments:

First of all, you don't use 1500 training sets, but one set with 1500 training patterns. The same goes for validation and testing.

Second, the output can't be exactly the same on each run, since the weights are initialized randomly and the outputs depend on them. However, we want them to be similar on each run. If they weren't it would mean that they depend too much on the random initialization, so the network wouldn't work well.

In your case, the highest output is the selected category, so if the same output is the highest every time your network is working well.

Answer 2

If the network output is almost the same for different input patterns, the network is unable to categorize input well.

You say your network has 4 input nodes and 20 output nodes (right?). So there are 2*2*2*2 = 16 different possible input patterns. Why the hell you need 800 validation sets?

Your training data may be corrupt.