CNN-LSTM in Keras: Dimension Error

Question

In keras 1.2.2, I have made a dataset which has these dimensions:

X_train: (2000, 100, 32, 32, 3)
y_train: (2000,1)

Here, 2000 is the number of instances (batches of data), 100 is the number of samples in each batch, 32 is the image rows and cols, and 3 is the number of channels (RGB).

I have written this code which applies an LSTM after a CNN. I used TimeDistributed layers and took the average of the output of LSTM to have something like this:

I want LSTM to work on each batch, and then I take the average of the LSTM output on that batch. So, my total output (my labels) is a (2000,1) vector.

I get this error:

Error when checking model target: expected lambda_14 to have 2 dimensions, but got array with shape (2000L, 100L, 1L)

This is my code:

# -*- coding: utf-8 -*-
# -*- coding: utf-8 -*-
import keras
from keras.layers import Input ,Dense, Dropout, Activation, LSTM
from keras.layers import Lambda, Convolution2D, MaxPooling2D, Flatten, Reshape
from keras.models import Sequential
from keras.layers.wrappers import TimeDistributed
from keras.layers.pooling import GlobalAveragePooling1D
from keras.optimizers import SGD
from keras.utils import np_utils
from keras.models import Model
import keras.backend as K


import numpy as np

timesteps=100;
number_of_samples=2500;
nb_samples=number_of_samples;
frame_row=32;
frame_col=32;
channels=3;

nb_epoch=1;
batch_size=timesteps;

data= np.random.random((2500,timesteps,frame_row,frame_col,channels))
label=np.random.random((2500,timesteps,1))

X_train=data[0:2000,:]
y_train=label[0:2000]

X_test=data[2000:,:]
y_test=label[2000:,:]

#%%

model=Sequential();                          

model.add(TimeDistributed(Convolution2D(32, 3, 3, border_mode='same'), input_shape=X_train.shape[1:]))
model.add(TimeDistributed(Activation('relu')))
model.add(TimeDistributed(Convolution2D(32, 3, 3)))
model.add(TimeDistributed(Activation('relu')))
model.add(TimeDistributed(MaxPooling2D(pool_size=(2, 2))))
model.add(TimeDistributed(Dropout(0.25)))

model.add(TimeDistributed(Flatten()))
model.add(TimeDistributed(Dense(512)))
#output dimension here is (None, 100, 512)                

model.add(TimeDistributed(Dense(35, name="first_dense" )))
#output dimension here is (None, 100, 35)                


model.add(LSTM(output_dim=20, return_sequences=True))
#output dimension here is (None, 100, 20)

time_distributed_merge_layer = Lambda(function=lambda x: K.mean(x, axis=1, keepdims=False))

model.add(time_distributed_merge_layer)
#output dimension here is (None, 1, 20)


#model.add(Flatten())
model.add(Dense(1, activation='sigmoid', input_shape=(None,20)))


model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

model.fit(X_train, y_train,
          batch_size=batch_size,
          nb_epoch=nb_epoch,
          validation_data=(X_test, y_test))

Nassim Ben Nassim Ben · Accepted Answer · 2017-03-02T11:30:53

If we follow the track of the shapes in your model, we get to shape = (None, 100, 35) after the Dense(). Then you feed that to an LSTM() which will return the whole sequence of hidden vectors of length 20 so you get a shape = (None, 100, 20). Then you take the mean over the axis 1 so you get shape = (None, 100, 1). There is something wrong with the architecture of that network because your targets have the shape = (None, 1). So either change

LSTM(output_dim=20, return_sequences=False)

and add a Flatten() layer after the merge. Or Flatten() after the merge and use a Dense(1, activation='sigmoid') to get your predictions?

It depends on you but right now it cannot work.

CNN-LSTM in Keras: Dimension Error

1 Answers