How can neural networks learn functions with a variable number of inputs?

6
votes

A simple example: Given an input sequence, I want the neural network to output the median of the sequence. The problem is, if a neural network learnt to compute the median of n inputs, how can it compute the median of even more inputs? I know that recurrent neural networks can learn functions like max and parity over a sequence, but computing these functions only requires constant memory. What if the memory requirement grows with the input size like computing the median?

This is a follow up question on How are neural networks used when the number of inputs could be variable?.

3
machine-learningneural-network
In your question sounds as if it is obvious that one can construct such a network that learns the median. Intuitively, I would say that this is not possible. – cel
Perhaps a better question is whether a Turing machine of fixed program size can calculate a median for an arbitrary number of inputs. If it can't then a neural network certainly can't. – dan
@cel If you use max and min pooling layers like CNNs do, I think the median problem is trainable. – Hanhan Li
@user3320467 Show me any fixed-size Turing machine and I can devise an input large enough to exceed your program's ability to handle all of it. My point was simple: variable input will require variable space. – dan
@user3320467 I understand what you're saying but how do you calculate the median without requiring variable space beyond the input size? – dan

3 Answers

1
votes

One idea I had is the following: treating each weight as a function of the number of inputs instead of a fixed value. So a weight may have many parameters that define a function, and we train these parameters. For example, if we want the neural network to compute the average of n inputs, we would like each weight function behaves like 1/n. Again, average per se can be computed using recurrent neural networks or hidden markov model, but I was hoping this kind of approaches can be generalized to solve certain problems where memory requirement grows.

1
votes

If a neural network learnt to compute the median of n inputs, how can it compute the median of even more inputs?

First of all, you should understand the use of a neural network. We, generally use the neural network in problems where a mathematical solution is not possible. In this problem, use of NN is not significant/ unadvisable.

There are other problems of such nature, like forecasting, in which continuous data arrives over time.

One solution to such problem can be Hidden Markov Model (HMM). But again, such models depends on the correlation between input over a period of time. So This model is not efficient for problems where the input is completely random.

So, If input is completely random and memory requirement grows

There is nothing much you can do about it, one possible solution could be growing your memory size.

Just remember one thing, NN and similar models of machine learning aims to extract meaningful information from the data. if data is just some random values then all models will generate some random output.

0
votes

One more idea: some data transformation. Let have N big enough that always bigger than n. We make a net with 2*N inputs. First N inputs are for data. If n less then N, then rest inputs set to 0. Last N inputs are intended for specifying which numbers are useful. Thus 1 is data, 0 is not data. As follows in Matlab notation: if v is an input, and it is a vector of length 2*N, then we put into v(1:n) our original data. After that, we put to v(n+1:N) zeros. Then put to v(N+1:N+n) ones, and then put V(N+n+1:2*N) zeros. It is just an idea, which I have not checked. If you are interested in the application of neural networks, take a look at the example of how we have chosen an appropriate machine learning algorithm to classify EEG signals for BCI.