variable scope issue in Tensorflow

Question

def biLSTM(data, n_steps):


    n_hidden= 24
    data = tf.transpose(data, [1, 0, 2])
    # Reshape to (n_steps*batch_size, n_input)
    data = tf.reshape(data, [-1, 300])
    # Split to get a list of 'n_steps' tensors of shape (batch_size, n_input)
    data = tf.split(0, n_steps, data)    

    lstm_fw_cell = tf.nn.rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)
    # Backward direction cell
    lstm_bw_cell = tf.nn.rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)

    outputs, _, _ = tf.nn.bidirectional_rnn(lstm_fw_cell, lstm_bw_cell, data, dtype=tf.float32)


    return outputs, n_hidden

In my code I am calling this function twice to create 2 bidirectional LSTMs. Then I got the problem of reusing variables.

ValueError: Variable lstm/BiRNN_FW/BasicLSTMCell/Linear/Matrix already exists, disallowed. Did you mean to set reuse=True in VarScope?

To resolve this I added the LSTM definition in the function within with tf.variable_scope('lstm', reuse=True) as scope:

This led to a new issue

ValueError: Variable lstm/BiRNN_FW/BasicLSTMCell/Linear/Matrix does not exist, disallowed. Did you mean to set reuse=None in VarScope?

Please help with a solution to this.

Mad Wombat Mad Wombat · Accepted Answer · 2017-01-10T20:18:41

When you create BasicLSTMCell(), it creates all the required weights and biases to implement an LSTM cell under the hood. All of these variables are assigned names automatically. If you call the function more than once within the same scope you get the error you get. Since your question seems to state that you want to create two separate LSTM cells, you do not want to reuse the variables, but you do want to create them in separate scopes. You can do this in two different ways (I haven't actually tried to run this code, but it should work). You can call your function from within a unique scope

def biLSTM(data, n_steps):
    ... blah ...

with tf.variable_scope('LSTM1'):
    outputs, hidden = biLSTM(data, steps)

with tf.variable_scope('LSTM2'):
    outputs, hidden = biLSTM(data, steps)

or you can pass a unique scope name to the function and use the scope inside

def biLSTM(data, n_steps, layer_name):
    ... blah...
    with tf.variable_scope(layer_name) as scope:
        lstm_fw_cell = tf.nn.rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)
        lstm_bw_cell = tf.nn.rnn_cell.BasicLSTMCell(n_hidden, forget_bias=1.0)
        outputs, _, _ = tf.nn.bidirectional_rnn(lstm_fw_cell, lstm_bw_cell, data, dtype=tf.float32)
    return outputs, n_hidden

l1 = biLSTM(data, steps, 'layer1')
l2 = biLSTM(data, steps, 'layer2')

It is up to your coding sensibilities which approach to choose, they are functionally pretty much the same.

variable scope issue in Tensorflow

2 Answers