Concatenate an input before Dense layer. [Keras with TF backend]

Question

So, I need to concatenate an input to the flattened layer before going in the dense layer. I'm using Keras with TF as backend.

model.add(Flatten())
aux_input = Input(shape=(1, )) 
model.add(Concatenate([model, aux_input])) 
model.add(Dense(512,kernel_regularizer=regularizers.l2(weight_decay)))

I have a scenario like this: X_train, y_train, aux_train. The shape of y_train and aux_train is same (1, ). An image has a ground-truth and an aux_input.

How do I add this aux_input to the model while doing model.fit?

As suggested in answers, I changed my model with functional api. However, now, I get the following error.

ValueError: Layer dense_1 was called with an input that isn't a symbolic tensor. Received type: . Full input: []. All inputs to the layer should be tensors.

Here's the code for that part.

flatten = Flatten()(drop_5)
aux_rand = Input(shape=self.aux_shape)
concat = Concatenate([flatten, aux_input])

fc1 = Dense(512, kernel_regularizer=regularizers.l2(weight_decay))(concat)

Shape of aux input

aux_shape = (1,)

And then calling the model as follow

(x_train, y_train), (x_test, y_test) = cifar10.load_data()
x_train = x_train.astype('float32')
x_test = x_test.astype('float32')

aux_rand = np.random.rand(y_train.shape[0])
model_inst = cifar10vgg()
x_train_input = Input(shape=(32,32,3))
aux_input = Input(shape=(1,))
model = Model(inputs=[x_train_input, aux_input], output=model_inst.build_model())
model.fit(x=[x_train, aux_rand], y=y_train, batch_size=batch_size, steps_per_epoch=x_train.shape[0] // batch_size,
                epochs=maxepoches, validation_data=(x_test, y_test),
                callbacks=[reduce_lr, tensorboard], verbose=2)

model_inst.build_model() returns Activation('softmax')(fc2) which is the output to be fed into the Model (as far as I understood)

you feed x_train (numpy array) into Model which is not correct! Inputs and outputs of the Model should be Tensor. — Amir
cifar10vgg() is a class which has a method build_model, that returns the output of the model. — nirvair
Updated the Model to use tensors (hopefully, that's the correct way to do it). Still getting an error — nirvair

Amir Amir · Accepted Answer · 2019-01-08T15:01:31

As I see from your code, you implement the model with sequential API which is not a good option in this case. If you have some auxiliary inputs the best way to implement such a feature is to use functional API.

Here is a example from Keras website:

from keras.layers import Input, Embedding, LSTM, Dense
from keras.models import Model

main_input = Input(shape=(100,), dtype='int32', name='main_input')

x = Embedding(output_dim=512, input_dim=10000, input_length=100)(main_input)

lstm_out = LSTM(32)(x)

auxiliary_output = Dense(1, activation='sigmoid', name='aux_output')(lstm_out)
auxiliary_input = Input(shape=(5,), name='aux_input')
x = keras.layers.concatenate([lstm_out, auxiliary_input])

x = Dense(64, activation='relu')(x)

main_output = Dense(1, activation='sigmoid', name='main_output')(x)

model = Model(inputs=[main_input, auxiliary_input], outputs=[main_output, auxiliary_output])

Based on description, I think following code can give you some intuition:

x1 = Input(shape=(32, 32, 3))
flatten1 = Flatten()(x1)

x2 = Input(shape=(244, 244, 3))
vgg = VGG19(weights='imagenet', include_top=False)(x2)
flatten2 = Flatten()(vgg)

concat = Concatenate()([flatten1, flatten2])
d = Dense(10)(concat)

model = Model(inputs=[x1, x2], outputs=[d])
model.compile('adam', 'categorical_crossentropy')
model.fit(x=[x_train1, x_train2],outputs=y_labels)

Concatenate an input before Dense layer. [Keras with TF backend]

1 Answers