stateful autoencoder in Keras

Question

I'm trying to create a stateful autoencoder model. The goal is to make the autoencoder stateful for each timeseries. The data consists of 10 timeseries and each timeseries has 567 length.

timeseries#1: 451, 318, 404, 199, 225, 158, 357, 298, 339, 155, 135, 239, 306, ....
timeseries#2: 304, 274, 150, 143, 391, 357, 278, 557, 98, 106, 305, 288, 325, ....
...
timeseries#10: 208, 138, 201, 342, 280, 282, 280, 140, 124, 261, 193, .....

My lookback windeow is 28. So I generated the following sequences with 28 timesteps:

[451, 318, 404, 199, 225, 158, 357, 298, 339, 155, 135, 239, 306, .... ]
[318, 404, 199, 225, 158, 357, 298, 339, 155, 135, 239, 306, 56, ....]
[404, 199, 225, 158, 357, 298, 339, 155, 135, 239, 306, 56, 890, ....]
...
[304, 274, 150, 143, 391, 357, 278, 557, 98, 106, 305, 288, 325, ....]
[274, 150, 143, 391, 357, 278, 557, 98, 106, 305, 288, 325, 127, ....]
[150, 143, 391, 357, 278, 557, 98, 106, 305, 288, 325, 127, 798, ....]
...
[208, 138, 201, 342, 280, 282, 280, 140, 124, 261, 193, .....]
[138, 201, 342, 280, 282, 280, 140, 124, 261, 193, 854, .....]

That gives me 539 sequences for each timeseries. What I need to do is to make the LSTMs to be stateful for each of the timeseries and reset the state after seeing all the sequences from a timeseries. Here is the code I have:

batch_size = 35  #(total Number of samples is 5390, and it is dividable by 35)
timesteps = 28
n_features = 1
hunits = 14
RepeatVector(timesteps/hunits = 2)
epochs = 1000


inputEncoder = Input(batch_shape=(35, 28, 1), name='inputEncoder')
outEncoder, c, h = LSTM(14, stateful=True, return_state=True, name='outputEncoder')(inputEncoder)
encoder_model = Model(inputEncoder, outEncoder)

context = RepeatVector(2, name='inputDecoder')(outEncoder)
context_reshaped = Reshape(28, 1), name='ReshapeLayer')(context)

outDecoder = LSTM(1, return_sequences=True, stateful=True, name='decoderLSTM')(context_reshaped)

autoencoder = Model(inputEncoder, outDecoder)

autoencoder.compile(loss='mse', optimizer='rmsprop')

for i in range(epochs):
       history = autoencoder.fit(data, data,
                          validation_split=config['validation_split_ratio'],
                          shuffle=False,
                          batch_size=35,
                          epochs=1,
                         )   
       autoencoder.reset_states()

2 questions:

1- I'm getting this error after the first epoch is finished, I wonder how it is happening:

ValueError: Cannot feed value of shape (6, 28, 1) for Tensor u'inputEncoder:0', which has shape '(35, 28, 1)'

2- I don't think that model works as I want. Here it will reset the states after all batches (one epoch) which means after all timeseries are processed. How should I change it to be stateful between timeseries?

Honestly I don't know what's going on with the first error. I think both problems go away if you write your own callback for resetting states. At the very end of this post you have something very similar to what you want philipperemy.github.io/keras-stateful-lstm. Hope this help. Another thing I don't understand is the number of samples: 538 sequences X 10 time series would be 5380 samples which is not divisible by 35. Is it an errata? — lsmor
@LuisMorillo Thanks for your comment, I just posted the solution. And 539 is the correct value, I'll fix the typo in the question. — Birish

Birish Birish · Accepted Answer · 2018-07-30T09:21:51

The issue is from the validation_split rate!! It is set to a 0.33% and when the splits happens it tries to train on 3611 data samples which is not divisible by my batch_size=35 . Based on this post I could find the proper number, copying from that post:

def quantize_validation_split(validation_split, sample_count, batch_size):
    batch_count = sample_count / batch_size
    return float(int(batch_count * validation_split)) / batch_count
then you can call model.fit(..., validation_split=fix_validation_split(0.05, len(X), batch_size)). but it would be cool if keras did this for you inside fit().

Also, regarding make the autoencoder stateful the way I need: there shouldn't be a reset_state at the end of each epoch!

stateful autoencoder in Keras

1 Answers