Keras: How to concatenate 2 tensors with dynamic shapes?

Question

I have a use case where I need to concatenate a 2D tensor to a 3D tensor in Keras. The dimensions of the 3D tensor are dynamic (for example, the 3D tensor could be the output of an LSTM layer with shape [batch_size, num_timesteps, num_features], where batch_size and num_timesteps are dynamic).

I have used the RepeatVector to repeat the values of the 2D tensor before the "merge" operation with the 3D tensor.

But, in my case, the "merge" operation throws an error (error details below). I've shared below a representative code for the operations I'm trying to achieve, along with the error.

I suspect the problem here is RepeatVector for a dynamic shape. Or, am I missing something more fundamental? Is there a way I can achieve this correctly?

I'm using Keras v2.1.6 with Tensorflow backend v1.8.0.

import keras
from keras.layers import *
input_3D = Input(shape=(None,100,), dtype='int32', name='input_3D')
input_2D = Input(shape=(100,), dtype='int32', name='input_2D')
input_2D_repeat = RepeatVector(K.shape(input_3D)[1])(input_2D)
merged = merge([input_3D, input_2D_repeat], name="merged", mode='concat')

The above code throws below error for the "merge" operation:

ValueError: "concat" mode can only merge layers with matching output shapes except for the concat axis. Layer shapes: [(None, None, 100), (None, , 100)]

I can see that the second dimension in input_3D is None, but the second dimension in input_2D_repeat is tf.Tensor 'strided_slice:0' shape=() dtype=int32.

How can I fix this?

jdehesa jdehesa · Accepted Answer · 2019-01-15T17:57:25

EDIT:

After reading again the question and my answer, I think the solution I posted is not correct. I think you meant to concatenate along the features axis, not the time axis, and also I'm not sure if tensor values like K.shape(input_3D)[1] can be used as parameters to a layer like RepeatVector. For your case, I think I would probably just resort to a Lambda layer to do the whole thing:

import keras
from keras.layers import Input, Lambda
import keras.backend as K

def repeat_and_concatenate(inputs):
    input_3D, input_2D = inputs
    # Repeat 2D vectors
    input_2D_repeat = K.tile(K.expand_dims(input_2D, 1), [1, K.shape(input_3D)[1], 1])
    # Concatenate feature-wise
    return K.concatenate([input_3D, input_2D_repeat], axis=-1)

input_3D = Input(shape=(None,100,), dtype='int32', name='input_3D')
input_2D = Input(shape=(50,), dtype='int32', name='input_2D')
merged = Lambda(repeat_and_concatenate)([input_3D, input_2D])
print(merged)
# Tensor("lambda_1/concat:0", shape=(?, ?, 150), dtype=int32)

THE FORMER ANSWER BELOW IS LIKELY WRONG

You need to specify that you want to concatenate along the "time" axis (instead of the default, which is the last one). You can do this:

import keras
from keras.layers import *

input_3D = Input(shape=(None,100,), dtype='int32', name='input_3D')
input_2D = Input(shape=(100,), dtype='int32', name='input_2D')
input_2D_repeat = RepeatVector(K.shape(input_3D)[1])(input_2D)
merged = Concatenate(axis=1)([input_3D, input_2D_repeat])

Or this:

import keras
from keras.layers import *

input_3D = Input(shape=(None,100,), dtype='int32', name='input_3D')
input_2D = Input(shape=(100,), dtype='int32', name='input_2D')
input_2D_repeat = RepeatVector(K.shape(input_3D)[1])(input_2D)
merged = concatenate([input_3D, input_2D_repeat], axis=1)

Keras: How to concatenate 2 tensors with dynamic shapes?

1 Answers