Concatenating embedded inputs to feed recurrent LSTM in keras

Question

I'm trying to figure out how to feed a functional model to LSTM gates in keras. I've got a time series of tuples (int, float, float). The ints are not ordered and should go through and embedding layer. I then want the tuples (after embedding the int) to go through an LSTM layer.

I've started with

from keras.layers import Input
from keras.layers import Embedding
from keras.layers import LSTM

import keras

inputs = [(42, 0.5, 0.6), (36, 0.4, 0.7), (50, 0.2, 0.9)] # example. The real data is a sequence of millions of tuples

input_id = Input(shape=(1,), dtype='int32', name='input_type') # id is within [0, 99]
embed_id = Embedding(output_dim=3, input_dim=20, input_length=1)(input_id)

input_v1 = Input(shape=(1,), dtype='float', name='input_v1')
input_v2 = Input(shape=(1,), dtype='float', name='input_v2')

input_merged = keras.layers.concatenate([embed_id, input_v1, input_v2], axis=-1)

lstm = LSTM(40) # how do I tell it to use input_merged as input ?

The concatenate complains: ValueError: Concatenate layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 1, 3), (None, 1), (None, 1)]

This I'm pretty sure could be arrange with reshapes. But What I'm really wondering: Is this the right approach to feeding Keras time sequences of data needing some processing ?

I'm also unsure how to feed the LSTM gates the concatenated results. All the recurrent examples I could find use sequential models.

Daniel Möller Daniel Möller · Accepted Answer · 2017-07-07T14:25:04

The embedding layer will need that you have the entire sequence of integers, so it can compute correctly.

So, as a first step, I'd split the input data:

intInputs = np.array([el[0] for el in inputs])    
floatInputs = np.array([el[1],el[2]] for el in inputs)
    #this could be optimized creating empty numpy arrays first and doing only one loop to fill them

After that, we must understand that if we're giving the entire sequence to the embedding, we must also pass the entire sequence to the model. For that, I think it's best to reshape the inputs to have "one sample" of many time steps (unless you really have more than one sequence):

intInputs = intInputs.reshape((1,seqLength))   
floatInputs = floatInputs.reshape((1,seqLength,2))

Then we go to the model:

input_id=Input((seqLength,))
input_v =Input((seqLength,2))

embed_id = Embedding(output_dim=3,input_dim=20,input_length=seqLength)(input_id)
    #are you sure "input_dim" is 20? Normally you should have the total amount of existing Ids, and I suppose this should at least be the last id+1.

With this, the concatenation will have shapes (None,seqLength,3), (None,seqLength,2), and you will be able to concat in the last axis (the others are equal in length).

input_merged = Concatenate(axis=-1)([embed_id,input_v])

And the LSTM receives the input as any other layer:

lstm = LSTM(40)(input_merged)

Concatenating embedded inputs to feed recurrent LSTM in keras

1 Answers