Keras: Embedding layer for multidimensional time steps

Question

I am trying to feed two sentences in character level into the LSTM layer for classification. My samples are similar to the following and my labels are one hot encoded classes.

label:

label array([1., 0., 0., 0., 0.])

sample:

array([['0', ' '],
   [' ', 'l'],
   ['1', 'e'],
   ['1', 't'],
   ['2', 'n'],
   ['8', 'i'],
   [' ', ' '],
   ['"', ';'],
   ['h', 'h'],
   ['t', 's'],
   ['t', 'o'],
   ['p', 't'],
   ['s', 'n'],
   [':', 'i'],
   ['/', 'c'],
   ['/', 'a'],
   ['w', 'm'],
   ['w', '('],
   ['w', ' '],
   ['.', '0'],
   ['e', '.'],
   ['x', '5'],
   ['a', '/'],
   ['m', 'a'],
   ['p', 'l'],
   ['l', 'l'],
   ['e', 'i'],
   ['.', 'z'],
   ['c', 'o'],
   ['o', 'm'],
   ['m', '"'],
   ['/', ' '],
   ['c', '"'],
   ['m', '/'],
   ['s', 'd'],
   ['/', 'a'],
   ['t', 'o'],
   ['i', 'l'],
   ['n', 'n'],
   ['a', 'w'],
   ['-', 'o'],
   ['a', 'd'],
   ['c', '-'],
   ['c', 'r'],
   ['e', 'o'],
   ['s', 'f'],
   ['s', '-'],
   ['-', 'r'],
   ['e', 'o'],
   ['d', 't'],
   ['i', 'i']], dtype='<U1')

I am trying to use the Embedding layer of Keras to map the characters into vectors. The embedding layer, however, only takes in single dimensional sequences. How can I adjust the network to take in multi dimensional sequence? Currently I have the following code that works for single dimensional samples. 51 is my lstm window size and 74 is the size of my vocabulary.

model = keras.models.Sequential()
model.add(keras.layers.Embedding(input_dim=74,
                                 output_dim=74,
                                 input_length=51))

model.add(keras.layers.Dropout(0.2))
model.add(keras.layers.LSTM(64,
                            dropout=0.5,
                            recurrent_dropout=0.5,
                            return_sequences=True,
                            input_shape=(51, 74)))
model.add(keras.layers.LSTM(64,
                            dropout=0.5,
                            recurrent_dropout=0.5))
model.add(keras.layers.Dense(num_classes, activation='sigmoid'))
model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

The Embedding does work with higher dimensions but the upstream LSTM won't. If you pass input_shape=(...) the Embedding should embed higher dimensional tensors as well. — nuric
I don't think that Embedding works for higher dimensions. From Keras documentation input_shape: input_dim: int > 0. Size of the vocabulary, i.e. maximum integer index + 1. — Fardin Abdi
input_dim is just the index size, has nothing to do with the shape of the actually tensor that is input. — nuric

Fardin Abdi Fardin Abdi · Accepted Answer · 2018-05-22T16:25:36

Ok, I solved this problem by adding a Reshaped layer before Embedding, and then another reshape layer after embedding. Here is the code:

model = keras.models.Sequential()

model.add(keras.layers.Reshape((2 * lstm_window_size, 1), input_shape=(
    lstm_window_size, 2)))

model.add(keras.layers.Embedding(input_dim=vocab_size + 1,
                                 output_dim=100,
                                 input_length=lstm_window_size * 2))

model.add(keras.layers.Reshape((lstm_window_size, 200)))
model.add(keras.layers.Dropout(0.2))
model.add(keras.layers.LSTM(64,
                            dropout=0.5,
                            recurrent_dropout=0.5,
                            return_sequences=True,
                            input_shape=(lstm_window_size, 2)))
model.add(keras.layers.LSTM(64,
                            dropout=0.5,
                            recurrent_dropout=0.5))
model.add(keras.layers.Dense(num_classes, activation='sigmoid'))
model.compile(loss='categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

Keras: Embedding layer for multidimensional time steps

1 Answers