Simple LSTM training with return_sequences=True in Keras

Question

I am trying to train a simple LSTM based RNN model in Keras using return_sequences. As I want all the timestep output. Here is the code I am using:

model = Sequential()
model.add(LSTM(10, return_sequences = True, input_shape = (8, 8)))
model.add(Activation('softmax'))

model.compile(loss = 'categorical_crossentropy', optimizer = adam, metrics = ['accuracy'])
model.fit(X_train, y_train, batch_size = 1, epochs = 1000, verbose = 1)

But I get the following error:

model.fit(X_train, y_train, batch_size = 1, epochs = 1000, verbose = 1)
File "C:\Program Files\Anaconda3\lib\site-packages\keras\models.py", line 1002, in fit
validation_steps=validation_steps)
File "C:\Program Files\Anaconda3\lib\site-packages\keras\engine\training.py", line 1630, in fit
batch_size=batch_size)
File "C:\Program Files\Anaconda3\lib\site-packages\keras\engine\training.py", line 1480, in _standardize_user_data
exception_prefix='target')
File "C:\Program Files\Anaconda3\lib\site-packages\keras\engine\training.py", line 113, in _standardize_input_data
'with shape ' + str(data_shape))
ValueError: Error when checking target: expected activation_1 to have 3 dimensions, but got array with shape (1437, 10)

But the same code works fine when I change the line from:

model.add(LSTM(10, return_sequences = True, input_shape = (8, 8)))

to:

model.add(LSTM(10, return_sequences = False, input_shape = (8, 8)))

I am assuming I will have to change the Activation('softmax') layer definition, but not sure how. Can someone help me in resolving this?

TIA

The shape of y_train is (1437, 10) and the output of the Activation layer is (None, 8, 10) when return_sequences=True. So, I understand it's the shape mismatch of Activation output shape and the y_train shape. I want return_sequences only after LSTM layer and not after activation layer. — Rachit Agrawal
Then why are you passing return_sequences=True? If you insist on doing it, you can first flatten the output of LSTM layer and then have a Dense layer with 10 units as the last layer which its activation function is softmax. — today
I want the output of each timestep from LSTM for debugging. Hence using return_sequences. — Rachit Agrawal

thushv89 thushv89 · Accepted Answer · 2018-09-06T11:30:04

I see two main issues in the code.

First, you're trying to get the output of the LSTM cell with a softmax activation? As return_sequences=True, the output shape will be [batch_size, time_steps, 10]. You probably need to use a TimeDistributed layer to make sure the activation layer can take in a time distributed output as TimeDistributed(Activation('softmax')).

Next, your y_train is of shape [1437,10] which clearly is not time distributed. To use the labels to optimize a model that outputs an output of shape [batch_size, time_steps, 10], your labels need to have the same shape. So, if the same label repeats for all time steps, I'd change your code as follows,

# Making y_train in to time distributed format, i.e. [1437,8,10]
y_train = y_train.reshape(-1,1,10)
y_train = np.repeat(y_train,8,axis=1)

model = Sequential()
model.add(LSTM(10, return_sequences = True, input_shape = (8, 8)))

# Making sure Activation layer is TimeDistributed
model.add(TimeDistributed(Activation('softmax')))

model.compile(loss = 'categorical_crossentropy', optimizer = Adam(), metrics = ['accuracy'])
model.fit(X_train, y_train, batch_size = 1, epochs = 10, verbose = 1)

Simple LSTM training with return_sequences=True in Keras

1 Answers