Keras multi-step LSTM batch train classification at each step

Question

Question

How to batch train a multi-step LSTM in Keras for single-label multi-class classificaiton, at each time-step for > 2 classes?

Current Error

Each target batch is a 3-dimensional array with shape (batch_size, n_time_steps, n_classes) but Keras expects a 2-dimensional array.

Example/Context

Suppose we have daily closing prices for N stocks and for each day and stock: m features and one of three actions "bought", "held", "sold". If there are 30 days' worth of data per stock we may train an LSTM to predict each action (on each day, for each stock) as follows.

For each batch of samples of size n << N, X_train will have a shape of (n, 30, m) i.e. n samples, 30 time-steps, and m features. After one-hot encoding "bought", "held", and "sold" Y_train will have a shape of (n, 30, 3), which is an array of 3-dimensions.

The problem is that Keras is giving an error due to expecting Y_train to be 2-dimensional.

Here is a code snippet:

n_time_steps = 30
n_ftrs = 700
n_neurons = 100
n_classes = 3
batch_size = 256
n_epochs = 500
model = Sequential()
model.add(LSTM(n_neurons, input_shape=(n_time_steps, n_ftrs)))
model.add(Dense(n_classes, activation='sigmoid'))
model.compile(loss='categorical_crossentropy', optimizer='adam', 
    metrics=['accuracy'])

for e in range(n_epochs):
  X_train, Y_train = BatchGenerator()
  # Y_train.shape = (256, 30, 3)
  model.fit(X_train, Y_train, batch_size=batch_size, nb_epoch=1)

Error

Error when checking target: expected dense_20 to have 2 dimensions, 
but got array with shape (256, 30, 3)

Can you try setting the input shape to first LSTM like (None, n_time_steps, n_ftrs) without setting return_sequences to True? — Novak

today today · Accepted Answer · 2018-10-18T10:59:13

If you take a look at the model.summary() output you would realize what the problem is:

Layer (type)                 Output Shape              Param #   
=================================================================
lstm_1 (LSTM)                (None, 100)               320400    
_________________________________________________________________
dense_74 (Dense)             (None, 3)                 303       
=================================================================
Total params: 320,703
Trainable params: 320,703
Non-trainable params: 0
_________________________________________________________________

As you can see the output shape of LSTM layer is (None, 100) which means that only the output of last timestep is returned. And as a result, the output shape of Dense layer is (None, 3) which means that it would classify the whole input timeseries (i.e. the whole 30-day data of a stock) in to one of 3 classes. This is not what you want. Rather you want to classify each timestep of the input timeseries. To make this happen, as @VegardKT suggested, you can pass return_sequences=True to the LSTM layer to have its output at each timestep. Let's look at the model.summary() output after this change:

Layer (type)                 Output Shape              Param #   
=================================================================
lstm_2 (LSTM)                (None, 30, 100)           320400    
_________________________________________________________________
dense_75 (Dense)             (None, 30, 3)             303       
=================================================================
Total params: 320,703
Trainable params: 320,703
Non-trainable params: 0
_________________________________________________________________

As you can see, now the LSTM layer gives the output of each timestep and therefore the Dense layer which acts a classifier would be able to classify each of those timesteps into one of 3 classes as desired.

Keras multi-step LSTM batch train classification at each step

2 Answers