CNTK Output Feedback

Question

I want to implement a CNTK Network that is able to learn a linear motion model. Similar to a Kalman Filter task the Network will receive measurement data from an accelerometer and should output a change in the current position.

dx = v_0 * dt + 1/2 * a * dt²

              _____                       __________
input:  a -> |     | -> dx          a -> |          | -> dx :output 
        v -> | NN  | -> dv               | v     dv | -> dv
        ^    |_____|     |               | ^      | |
        |                |               | |_LSTM_| |
        |__ cumsum(dv)___|               |__________|

                                        Desired Aproach

Training Data for dv and dx is available.

Since the network should learn model free I do not want to input any motion model. My problem is that without memorizing the velocity from the previous time step the acceleration value is not directly linked to the position change.

Or do I have to do something like this:

              _____                               _____
input:  a -> |     | -> dv -> cumsum(dv) -> v -> |     | -> dx
             | NN  |                             | NN  |
             |_____|                             |_____|

Unfortunately Tutorials and Examples do not contain a similar task. I think it has to work with a RNN, Elman or Jordan Network. But I am not able to implement that in python in CNTK. Has anyone done something similar before? I'm still a bit confused with dynamic and static axis and forward declaration.

Another formulation for my question would be: How to teach a network to integrate or sum.

Thanks for every comment or hint.

Nikos Karampatziakis Nikos Karampatziakis · Accepted Answer · 2017-07-18T01:56:20

Since it was not clear to me whether you want to be summing the dv or not, I will describe the case where you want v = cumsum(dv); you can easily replace that with a learnable function. I'm assuming 3-d acceleration data.

hidden_dim = 100
tradeoff = 0.5

accel = C.sequence.input_variable(3)
dv = C.sequence.input_variable(3)
dx = C.sequence.input_variable(3)

predicted_dv = C.layers.Recurrence(C.layers.LSTM(3, hidden_dim))(accel)
predicted_v = C.layers.Recurrence(C.plus)(predicted_dv)
combined_inputs = C.splice(accel, predicted_v)
predicted_dx = C.layers.Recurrence(C.layers.LSTM(3, hidden_dim))(combined_inputs)

loss = C.squared_error(predicted_dx, dx) + tradeoff * C.squared_error(predicted_dv, dv)

I arbitrarily chose the LSTMs to have 100 dimensional internal representation of state but output 3 dimensional output. I also arbitrarily chose to way the error on dx twice as much as the error on dv. The cumulative sum is a simple recurrence with plus as your "recurrent cell". You can change that to an LSTM if you want. To feed data, just provide a list of arrays of shape (time_steps, 3) where time_steps can be different for each array. If all time_steps are equal you can also feed a tensor of shape (batch, time_steps, 3)

CNTK Output Feedback

1 Answers