Input Permutations in Feed-Forward Neural Networks

Question

Given a feed-forward neural-network, how to:

1. Ensure that it is independent on the order of the inputs? e.g., feeding [0.2, 0.3] would output the same result as [0.3, 0.2];
2. Ensure that it is independent on the order of groups of inputs? e.g., feeding [0.2, 0.3, 0.4, 0.5] would output the same result as [0.4, 0.5, 0.2, 0.3], but not [0.5, 0.4, 0.3, 0.2];
3. Ensure that a permutation on the input sequence would give a permutation on the output sequence. e.g., if [0.2, 0.3] gives as output [0.8, 0.7], then [0.3, 0.2] gives as output [0.7, 0.8].

---

Given the above:

1. Is there any other solution besides ensuring that the train set covers all the possible permutations?
2. Is the parity of the hidden layer somehow constrained (i.e., the number of neurons in the hidden layer must be odd or even)?
3. Does it make sense too look for some sort of symmetry in the weight matrix?

Answer 1

well, it looks like a hard job for NN but 1. I'd make some preprocessing and maybe postprocessing script which would take care of all your permutation, make sure that the easiest possible input is given to NN. I think pre(post)processing would be much easier to achieve your goal than adjusting NN (adding one or more hidden layers)

2&3 NN are usually perceived as blackboxes. It means you train it and analyse just input and output. In most cases it doesn't make sense(time-demanding) to try to understand how is it working inside (of course there are some exceptions eg if you have functional NN and you would like to mine some knowledge - butas i said - it is time-consuming).

In general, there are no constraints regarding to number of hidden neurons per layer. Also, looking for symetry in weight matrix doesn't make sense unless you are trying to find some knowledge ...

Answer 2

Here is my try to answer the questions as best as i can

How to

1. To get the required results you can either

   • train all permutations
   • sort the input data and train it (so it doesn't have to learn the permutations extra)

2. To get the requested result you do have again two possibilities

   • train all permutations (timeconsuming)
   • or better, use another type of network, for example a recurrent neural network with the echo state network training algorithm (paper here)

3. i would try to solve it again with the echo state network algorithm

I hope it helps even if the possible solutions for the second and third problem are no feed forward networks.

Answering the questions

3 I don't think that it makes any sense to look for symetries in the weight matrix.