How to stack multiple lstm in keras?

Question

I am using deep learning library keras and trying to stack multiple LSTM with no luck. Below is my code

model = Sequential()
model.add(LSTM(100,input_shape =(time_steps,vector_size)))
model.add(LSTM(100))

The above code returns error in the third line Exception: Input 0 is incompatible with layer lstm_28: expected ndim=3, found ndim=2

The input X is a tensor of shape (100,250,50). I am running keras on tensorflow backend

Answer 1

You need to add return_sequences=True to the first layer so that its output tensor has ndim=3 (i.e. batch size, timesteps, hidden state).

Please see the following example:

# expected input data shape: (batch_size, timesteps, data_dim)
model = Sequential()
model.add(LSTM(32, return_sequences=True,
               input_shape=(timesteps, data_dim)))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32, return_sequences=True))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32))  # return a single vector of dimension 32
model.add(Dense(10, activation='softmax'))

From: https://keras.io/getting-started/sequential-model-guide/ (search for "stacked lstm")

Answer 2

Detail explanation to @DanielAdiwardana 's answer. We need to add return_sequences=True for all LSTM layers except the last one.

Setting this flag to True lets Keras know that LSTM output should contain all historical generated outputs along with time stamps (3D). So, next LSTM layer can work further on the data.

If this flag is false, then LSTM only returns last output (2D). Such output is not good enough for another LSTM layer.

# expected input data shape: (batch_size, timesteps, data_dim)
model = Sequential()
model.add(LSTM(32, return_sequences=True,
               input_shape=(timesteps, data_dim)))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32, return_sequences=True))  # returns a sequence of vectors of dimension 32
model.add(LSTM(32))  # return a single vector of dimension 32
model.add(Dense(10, activation='softmax'))

On side NOTE :: last Dense layer is added to get output in format needed by the user. Here Dense(10) means 10 different classes output will be generated using softmax activation.

In case you are using LSTM for time series then you should have Dense(1). So that only one numeric output is given.

Answer 3

An example code like this should work:

regressor = Sequential()

regressor.add(LSTM(units = 50, return_sequences = True, input_shape = (33, 1)))
regressor.add(Dropout(0.2))

regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))

regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.2))

regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.2))

regressor.add(Dense(units = 1))

regressor.compile(optimizer = 'adam', loss = 'mean_squared_error')

regressor.fit(X_train, y_train, epochs = 10, batch_size = 4096)