Keras Function (K.function) not working with RNN (code provided)

Question

I was trying to see the each of the layers' outputs on Keras, but I couldn't get to the correct code so I made a simple code where I was stuck at.

Question: How am I supposed to get the output of each layer where there is RNN layer in the entire layers?

You can see how I tried to see in the below code.

Here's the test code that is working(1):

seq_length = 3
latent_dim = 2
inputs = Input(shape=(seq_length, latent_dim))
outputs = Dense(5)(inputs)
outputs = Flatten()(outputs)

model = Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='rmsprop', loss='mse')
print(model.summary())

To see the output of each layer(2):

layer_outputs = list()
for idx, l in enumerate(model.layers):
    if idx == 0:
        continue
    layer_outputs.append(l.output)
get_3rd_layer_output = K.function([model.layers[0].input],
                                  layer_outputs)
layer_output = get_3rd_layer_output([enc_input])
print('')
for l_output in layer_output:
    print(l_output[0][0])
    print('')

then the output would be something like

[ 4.172303 -2.248884 1.397713 3.2669916 2.5788064]

4.172303

However, if I try to test the same logic as (2) with below code that uses RNN:

seq_length = 3
latent_dim = 2
inputs = Input(shape=(seq_length, latent_dim))
outputs, last_output = GRU(latent_dim, return_state=True, return_sequences=True)(inputs)

model = Model(inputs=inputs, outputs=outputs)
model.compile(optimizer='rmsprop', loss='mse')
print(model.summary())

and test with (2) it will emit like as follows:

--------------------------------------------------------------------------- TypeError Traceback (most recent call last) in 5 layer_outputs.append(l.output) 6 get_3rd_layer_output = K.function([model.layers[0].input], ----> 7 layer_outputs) 8 layer_output = get_3rd_layer_output([enc_input]) 9 print('')

d:\igs_projects\nlp_nlu\venv\lib\site-packages\keras\backend\tensorflow_backend.py in function(inputs, outputs, updates, **kwargs) 2742
msg = 'Invalid argument "%s" passed to K.function with TensorFlow backend' % key 2743 raise ValueError(msg) -> 2744 return Function(inputs, outputs, updates=updates, **kwargs) 2745 2746

d:\igs_projects\nlp_nlu\venv\lib\site-packages\keras\backend\tensorflow_backend.py in init(self, inputs, outputs, updates, name, **session_kwargs)
2544 self.inputs = list(inputs) 2545 self.outputs = list(outputs) -> 2546 with tf.control_dependencies(self.outputs): 2547 updates_ops = [] 2548 for update in updates:

d:\igs_projects\nlp_nlu\venv\lib\site-packages\tensorflow\python\framework\ops.py in control_dependencies(control_inputs) 5002 return _NullContextmanager() 5003 else: -> 5004 return get_default_graph().control_dependencies(control_inputs) 5005
5006

d:\igs_projects\nlp_nlu\venv\lib\site-packages\tensorflow\python\framework\ops.py in control_dependencies(self, control_inputs) 4541 if isinstance(c, IndexedSlices): 4542 c = c.op -> 4543 c = self.as_graph_element(c) 4544 if isinstance(c, Tensor): 4545 c = c.op

d:\igs_projects\nlp_nlu\venv\lib\site-packages\tensorflow\python\framework\ops.py in as_graph_element(self, obj, allow_tensor, allow_operation) 3488 3489 with self._lock: -> 3490 return self._as_graph_element_locked(obj, allow_tensor, allow_operation) 3491 3492 def _as_graph_element_locked(self, obj, allow_tensor, allow_operation):

d:\igs_projects\nlp_nlu\venv\lib\site-packages\tensorflow\python\framework\ops.py in _as_graph_element_locked(self, obj, allow_tensor, allow_operation) 3577 # We give up! 3578 raise TypeError("Can not convert a %s into a %s." % (type(obj).name, -> 3579 types_str)) 3580 3581 def get_operations(self):

TypeError: Can not convert a list into a Tensor or Operation.

Manoj Mohan Manoj Mohan · Accepted Answer · 2019-04-30T10:58:46

For the GRU layer, layer.output is itself a list.

>>> model.layers[1].output
[<tf.Tensor 'gru_1/transpose_1:0' shape=(?, ?, 2) dtype=float32>, <tf.Tensor 'gru_1/while/Exit_3:0' shape=(?, 2) dtype=float32>]

layer_outputs is a list containing another list hence the error "Can not convert a list into a Tensor or Operation".

>>> layer_outputs
[[<tf.Tensor 'gru_1/transpose_1:0' shape=(?, ?, 2) dtype=float32>, <tf.Tensor 'gru_1/while/Exit_3:0' shape=(?, 2) dtype=float32>]]

Updating the code like this should work:

get_3rd_layer_output = K.function([model.layers[0].input],
                                  layer_outputs[0]) #Extract the element and feed it.

Keras Function (K.function) not working with RNN (code provided)

1 Answers