Keras - Trying to concatenate two inputs

Question

I have two inputs to my model, both of which are tensors (one is an input layer, and the other is a embedding layer). I am using the concatenate which is for tensors and not Concatenate which is for layers. I have does this before without issue, but I currently am using a different dataset where the inputs have a different shape. What I am trying to do is concatenate an image with an embedding matrix and pass it into a densenet121:

-----------
|embedding|
|         |
-----------
|  image  |
-----------

Here are their original shapes:

Image: (?, 224, 224, 1)
embedding: (?, 200, 224)

Clearly they are of different size (one is a square and one is more of a rectangle) and have different number of dims. So I tried to concatenate as follows:

merged = Concatenate([text_embedding, squeeze(image_input, axis=-1)], axis=1, name='merged')

The reasoning behind the squeeze is because it is of shape (?, 224, 224, 1) and embedding is as shown above. I suspect that It could have to either be 1 of 2 things:

The concat axis is wrong
The concat function cant operate on these inputs (as they may be layers, must use Concat?)
Maybe both shapes must have 4 dims?

I have received the following errors:

default:

ValueError: Shape must be rank 4 but is rank 3 for 'sequential_11/densenet121/zero_padding2d_21/Pad' (op: 'Pad') with input shapes: [?,424,224], [4,2].

for 1) I tried setting the concat axis to 2 and got:

ValueError: A `Concatenate` layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 200, 224, 1), (None, 224, 224, 1)]

for 2) changed concat to Concat TypeError: __init__() got multiple values for argument 'axis'

for 3) I tried: expand_dims(text_embedding, axis=-1) and got: AttributeError: 'NoneType' object has no attribute '_inbound_nodes'

Any idea how I can fix this?

Kevin Kevin · Accepted Answer · 2020-01-14T03:42:21

Okay, found my answer:

expandedText = Lambda(lambda x: expand_dims(x, axis=-1))
merged = concatenate([expandedText(text_embedding), image_input], axis=1, name='merged')

Apparently you have to turn it into a layer before making the graph as found here.

Keras - Trying to concatenate two inputs

1 Answers