TensorFlow (v1.1.0) Multi-RNN BasicLSTMCell Error ('reuse' parameter) Python 3.5

Question

An expansion on: What is the use of a "reuse" parameter of tf.contrib.layers functions?.

The question: Although this issue has been brought up on github and will likely be addressed in another release of TensorFlow, I have found no existing solution for the time being; is there a stop-gap measure that might work in the meantime?

Code:

state_size = 4
def lstm_cell():
    if 'reuse' in inspect.getargspec(tf.contrib.rnn.BasicLSTMCell.__init__).args:
        return tf.contrib.rnn.BasicLSTMCell(state_size, forget_bias=0.0, state_is_tuple=True, reuse=tf.get_variable_scope().reuse)
    else:
        return tf.contrib.rnn.BasicLSTMCell(state_size, forget_bias=0.0, state_is_tuple=True)

cell = lstm_cell()
cell = rnn.DropoutWrapper(cell, output_keep_prob=0.5)
cell = rnn.MultiRNNCell([cell] * num_layers, state_is_tuple=True)
states_series, current_state = tf.nn.dynamic_rnn(cell, tf.expand_dims(batchX_placeholder, -1), initial_state=rnn_tuple_state)
states_series = tf.reshape(states_series, [-1, state_size])

The function lstm_cell() is a suggestion from https://github.com/tensorflow/models/blob/master/tutorials/rnn/ptb/ptb_word_lm.py. It explains that the newest version of tensorflow has includes the 'reuse' parameter for BasicLSTMCell().

In this code, if I set reuse to False, tf.nn.dynamic_rnn line produces the error:

"ValueError: Variable rnn/multi_rnn_cell/cell_0/basic_lstm_cell/weights already exists, disallowed. Did you mean to set reuse=True in VarScope? Originally defined at:..."

If I set reuse to True, the error is:

"ValueError: Attempt to reuse RNNCell with a different variable scope than its first use. First use of cell was with scope 'rnn/multi_rnn_cell/cell_0/basic_lstm_cell', this attempt is with scope 'rnn/multi_rnn_cell/cell_1/basic_lstm_cell'. Please create a new instance of the cell if you would like it to use a different set of weights. If before you were using: MultiRNNCell([BasicLSTMCell(...)] * num_layers), change to: MultiRNNCell([BasicLSTMCell(...) for _ in range(num_layers)]). If before you were using the same cell instance as both the forward and reverse cell of a bidirectional RNN, simply create two instances (one for forward, one for reverse). In May 2017, we will start transitioning this cell's behavior to use existing stored weights, if any, when it is called with scope=None (which can lead to silent model degradation, so this error will remain until then.)"

Lastly, adding in 'scope=None' to dynamic_rnn provides no difference either.

dv3 dv3 · Accepted Answer · 2017-05-14T09:20:51

Have you considered trying what the 'reuse to True'-error is suggesting?

If before you were using: MultiRNNCell([BasicLSTMCell(...)] * num_layers), change to: MultiRNNCell([BasicLSTMCell(...) for _ in range(num_layers)]).

following code snipped works for me (already answered here)

def lstm_cell():
    cell = tf.contrib.rnn.NASCell(state_size, reuse=tf.get_variable_scope().reuse)
    return tf.contrib.rnn.DropoutWrapper(cell, output_keep_prob=0.8)

rnn_cells = tf.contrib.rnn.MultiRNNCell([lstm_cell() for _ in range(num_layers)], state_is_tuple = True)
outputs, current_state = tf.nn.dynamic_rnn(rnn_cells, x, initial_state=rnn_tuple_state)

TensorFlow (v1.1.0) Multi-RNN BasicLSTMCell Error ('reuse' parameter) Python 3.5

1 Answers