Sequential Neural Network

Question

I am trying to construct a neural network as a generative model, to predict the next vector following a sequence of vectors (each vector is a distribution of real numbers of length n).

My thought was to take k previous sequences and concatenate them to have a kxn input vector. To train the model, I would have the next vector in the sequence as the output. As I am looking for non-deterministic output, I was going to use a sigmoid activation function with low gradient.

Does this procedure seem reasonable?

In the hope it does, I tried implementing it in R using both the nnet and neuralnet libraries, but it the documentation and examples I came across, it seems the input and output vectors must be of the same length. What is the syntax to train on input/output vectors of varying length in either of those modules?

A sample of my input vector is:

      [,1]      
 [1,] 0         
 [2,] 0         
 [3,] 0.6       
 [4,] 0.4       
 [5,] 0         
 [6,] 0         
 [7,] 0.06666667
 [8,] 0.6666667 
 [9,] 0         
[10,] 0.2666667 
[11,] 0         
[12,] 0.4       
[13,] 0         
[14,] 0         
[15,] 0.6       

And output vector:

      [,1]    
 [1,] 0         
 [2,] 0         
 [3,] 0.8571429 
 [4,] 0         
 [5,] 0.1428571 

N.B. The above sample has n=5, k=3, although my actual dataset has n~200. In both cases, the individual vectors are normalized to 1.

Any help is much appreciated!

Answer 1

In general this is very simple and naive approach, which rather won't yield good results. Your are trying to perform the regression from set of time series into time series treating everything as simple attributes and simple model. There have been thousands of papers/research regarding time series predictions, representing time dependence etc.You are facing a hard type of prediction problem here, finding the good solution will require lots of work, and proposed model has a very little chance of working well.

From your text I deduce, that you actually have a sequence of time series, and for the "time window" [t-k,t-k+1,..,t-1] you want to predict the value (time series) in t. If this is true, then this is actualy time series prediction problem, where each attribute is the time series on its own, and all time series related techniques can be used here, as for example recurrent neural networks (if you really like NNs) or conditional RBMs (if you really want a non-deterministic, generative model - as they have been succesfully applied to time series prediction in recent years).

Now few other doubts:

As I am looking for non-deterministic output, I was going to use a sigmoid activation function

Sigmoid activation function is not non-deterministic. If you are looking for non deterministic models you should think about some architectures like RBMs, but as @Ben Allison mentioned in the comment, traditional neural networks can also be used in the probabilistic fashion with some simple modifications.

with low gradient.

What do you mean by low gradient? That your activation function has a small slope? This will result in a problematic learning in case of simple training procedures (like BP algorithm)

[DATA]

Your data looks like you normalized each time series so it sums to 1 which is rather not popular approach to data normzliazation in neural networks (you rather normalize data column-wise, so each dimension is normalized, not each sample).

Title

Your question, and model is not "sequentional" and does not include "varying vector lengths", looking for papers about such phenomena won't lead you to answer for your question.