How to convert 1D flattened MNIST Keras to LSTM model without unflattening?

Question

I want to change my model architecture a bit on the LSTM so it accepts the same exact flattened inputs the full connected approach does.

Working Dnn model from Keras examples

import keras

from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.utils import to_categorical

# import the data
from keras.datasets import mnist

# read the data
(x_train, y_train), (x_test, y_test) = mnist.load_data()

num_pixels = x_train.shape[1] * x_train.shape[2] # find size of one-dimensional vector

x_train = x_train.reshape(x_train.shape[0], num_pixels).astype('float32') # flatten training images
x_test = x_test.reshape(x_test.shape[0], num_pixels).astype('float32') # flatten test images

# normalize inputs from 0-255 to 0-1
x_train = x_train / 255
x_test = x_test / 255

# one hot encode outputs
y_train = to_categorical(y_train)
y_test = to_categorical(y_test)

num_classes = y_test.shape[1]
print(num_classes)



# define classification model
def classification_model():
    # create model
    model = Sequential()
    model.add(Dense(num_pixels, activation='relu', input_shape=(num_pixels,)))
    model.add(Dense(100, activation='relu'))
    model.add(Dense(num_classes, activation='softmax'))


    # compile model
    model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
    return model


# build the model
model = classification_model()

# fit the model
model.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=10, verbose=2)

# evaluate the model
scores = model.evaluate(x_test, y_test, verbose=0)

Same problem but trying LSTM (syntax error still)

def kaggle_LSTM_model():
    model = Sequential()
    model.add(LSTM(128, input_shape=(x_train.shape[1:]), activation='relu', return_sequences=True))
    # What does return_sequences=True do?
    model.add(Dropout(0.2))

    model.add(Dense(32, activation='relu'))
    model.add(Dropout(0.2))

    model.add(Dense(10, activation='softmax'))

    opt = tf.keras.optimizers.Adam(lr=1e-3, decay=1e-5)
    model.compile(loss='sparse_categorical_crossentropy', optimizer=opt,
             metrics=['accuracy'])

    return model

model_kaggle_LSTM = kaggle_LSTM_model()

# fit the model
model_kaggle_LSTM.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=10, verbose=2)

# evaluate the model
scores = model_kaggle_LSTM.evaluate(x_test, y_test, verbose=0)

Problem is here:

model.add(LSTM(128, input_shape=(x_train.shape[1:]), activation='relu', return_sequences=True))

ValueError: Input 0 is incompatible with layer lstm_17: expected ndim=3, found ndim=2

If I go back and don't flatten x_train and y_train, it works. However, I'd like this to be "just another model choice" that feeds off the same pre-processed input. I thought passing shape[1:] would work as that it the real flattened input_shape. I'm sure it's something easy I'm missing about the dimensionality, but I couldn't get it after an hour of twiddling and debugging, although did figure out not flattening the 28x28 to 784 works, but I don't understand why it works. Thanks a lot!

For bonus points, an example of how to do either DNN or LSTM in either 1D (784,) or 2D (28, 28) would be the best.

today today · Accepted Answer · 2019-03-26T17:33:05

RNN layers such as LSTM are meant for sequence processing (i.e. a series of vectors which their order of appearance matters). You can look at an image from top to bottom, and consider each row of pixels as a vector. Therefore, the image would be a sequence of vectors and can be fed to the RNN layer. Therefore, according to this description, you should expect that the RNN layer take an input of shape (sequence_length, number_of_features). That's why when you feed the images to the LSTM network in their original shape, i.e. (28,28), it works.

Now if you insist on feeding the LSTM model the flattened image, i.e. with shape (784,), you have at least two options: either you can consider this as a sequence of length one, i.e. (1, 748), which does not make much sense; or you can add a Reshape layer to your model to reshape back the input to its original shape suitable for the input shape of a LSTM layer, like this:

from keras.layers import Reshape

def kaggle_LSTM_model():
    model = Sequential()
    model.add(Reshape((28,28), input_shape=x_train.shape[1:]))
    # the rest is the same...

How to convert 1D flattened MNIST Keras to LSTM model without unflattening?

Working Dnn model from Keras examples

Same problem but trying LSTM (syntax error still)

1 Answers