Tensorflow - Keras disconnected graph

2
votes

Tensorflow Version: 2.x

Python: 3.7.4

Disconnected graph: I am trying to replicate the below model architecture, but right part seems to be disconnected when I tried to plot the model in Keras. I already passed the hidden matrices HQ(For question) and HA(For answer) as inputs to attention layer (We can see the inputs to Coattention layer in summary below - The input shapes are (512,600) and (512, 600) and Coattention output shapes are also same for matrices CQ and CA) . Please help me understand this disconnection. Does this needs to be corrected or can this be ignored?

Final Model:

inputs = [input_text1, input_text2]
outputs = score_oq_oa
model = Model(inputs=inputs, outputs=outputs)

model.summary()

enter image description here

Expected Model Architecture: enter image description here

Model generated graph: Why is it disconnected on right side? Please help me understand. I did not use concatenate layer after bidirectional layers of question and answer but I just passed output matrices of both bidirectional layers as inputs to attention layer as stated above. enter image description here

Question updated with code for Coattention layer as below:

Here HQ and HA are hidden state matrices/outputs of two separate bidirectional layers as we see in model architecture. 

class coattention(tf.keras.layers.Layer):

    def __init__(self):
        super(coattention, self).__init__()

    def call(self, HQ, HA):  

        L = tf.linalg.matmul(HA, HQ, transpose_a = True, transpose_b = False)
        AQ = tf.nn.softmax(L, axis = 1)
        AA = tf.nn.softmax(tf.transpose(L), axis = 1)

        CQ = tf.linalg.matmul(HA, AQ, transpose_a = False, transpose_b = False)
        CA = tf.linalg.matmul(HQ, AA, transpose_a = False, transpose_b = False)

        return CQ, CA


coattention_layer = coattention()
CQ, CA = coattention_layer(HQ, HA)
print ("Shape of Context vector of Question (CQ): ", CQ.shape)
print ("Shape of Context vector of Answer   (CA): ", CA.shape)

Shape of Context vector of Question (CQ): (512, 600)

Shape of Context vector of Answer (CA): (512, 600)

1
pythontensorflowkerastensorflow2.0question-answering
don't think it should be disconnected, mind sharing your code on the model definition (especially on the coattention layer) ? – emilyfy
@emilyfy - I added the code now for attention layer in question. Could you please take a look at it now. My Coattention layer produces/returns CA and CQ, so I have both inputs HA and HQ in a single line. But somehow in graph, it is showing only one connection on left side but not on right. – Raghu
@Ronakrit's comment is right, I usually wrap my inputs in a list if there's multiple variables. You can try defining it as def call(self, inputs): HQ, HA = inputs[0], inputs[1] – emilyfy
@emilyfy - It worked. Thank you! – Raghu

1 Answers

1
votes

Because you did not provide the code, I believe you forgot to call the coattention layer with Bidirectional_7 layer as input.

Here is the example code

Ha = Input(shape=(1,2,3), name='Ha')
Hq = Input(shape=(1,2,3), name='Hq')

your_coattention_layer = Dense(12, name='your_coattention_layer')

# this part that I think you forgot
Ca = your_coattention_layer(Ha)
cQ = your_coattention_layer(Hq)


out1 = Dense(123, name='your_Ca_layer')(Ca)
out2 = Dense(123, name='your_Cq_later')(cQ)
M = Model(inputs=[Ha,Hq], outputs=[out1,out2])
M.summary()

from keras.utils import plot_model
plot_model(M, to_file='Example.png')

This is the model picture.

enter image description here